Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining

Abstract: LLMs exhibit impressive cross-lingual capabilities. However, prior work analyzes this phenomenon through isolated factors and at sparse points during training, limiting our understanding of how cross-lingual generalization emerges--particularly in the early phases of learning. To study the early trajectory of linguistic and translation capabilities, we pretrain a multilingual 1.7B model on nine diverse languages, capturing checkpoints at a much finer granularity. We further introduce a novel word-level translation dataset and trace how translation develops over training through behavioral analyses, model-component analysis, and parameter-based ablations. We find that the model quickly acquires basic linguistic capabilities in parallel with token-level copying, while translation develops in two distinct phases: an initial phase dominated by copying and surface-level similarities, and a second phase in which more generalizing translation mechanisms are developed while copying is refined. Together, these findings provide a fine-grained view of how cross-lingual generalization develops during multilingual pretraining.

• The paper demonstrates that early pretraining rapidly induces token copying, achieving >90% accuracy in monolingual tasks before transitioning to translation.
• It employs dense checkpointing, activation probing, and parameter ablations to disentangle copying behaviors from genuine cross-lingual generalization.
• The study reveals that intermediate transformer layers develop robust semantic mappings, crucial for suppressing copying and enabling translation for non-overlapping languages.

Interpreting the Dynamics of Cross-Lingual Representation: Copy First, Translate Later

Introduction

The paper "Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining" (2604.17633) provides a systematic investigation into the sequential process by which cross-lingual generalization—particularly translation—emerges during multilingual LLM pretraining. The study leverages a 1.7B parameter LLaMA-style decoder-only model trained across nine typologically diverse languages and introduces a controlled, fine-grained word-level translation (WLT) dataset. It combines dense checkpointing throughout early training, behavioral metrics, activation probing, and parameterized ablations, enabling dissection of the intertwined dynamics of copying versus translation mechanisms. The work addresses an essential but insufficiently explored question: how do LLMs transition from relying on surface-level token copying to learning generalizable semantic mappings across languages, especially in low-resource or non-overlapping vocabularies?

Pretraining Protocol and Evaluation Framework

A 1.7B parameter LLaMA-architecture model is pretrained from scratch on nine languages representing Latin, Han, and Japanese scripts. Training leverages FineWeb-HQ and FineWeb2-HQ corpora, with English at 50% and each other language at 6.25% sampling probability per sequence. Critically, 200 checkpoints are captured at 185M-token intervals, resulting in an unprecedentedly dense pretraining trajectory. The evaluation protocol employs two axes: (1) linguistic acceptability via multi-lingual BLiMP-style minimal pairs, and (2) a bespoke WLT testbed spanning all 72 unidirectional language pairs, curated for PoS and lemma/frequency stratification and explicit synonym lists to model polysemous translation.

Two-Phase Emergent Dynamics in Multilingual Representation

Phase I: Rapid Copying Induction

Early training yields an immediate escalation in both monolingual acceptability (BLiMP tasks) and repetition (source-to-source copy) accuracy, with the model converging to >90% repetition accuracy and >90% BLiMP accuracy by 9.5B tokens. In this phase, word-level translation is dominated by naive token copying (peaking above 53% of errors), with both context copying (overexamples in prompt) and source word copying as the predominant error types. This immediate emergence of copying is consistent with the dominance of token overlap and the inductive bias of transformer architectures when exposed to ambiguous multilingual supervision.

Figure 1: Translation accuracy over model training, grouped by target language.

Phase II: Suppression of Copying and Generalizing Translation

After ~11B tokens, successful translations begin to emerge for word pairs lacking token overlap, marking a transition. Here, erroneous copying steadily declines and the model increasingly deploys generalizing translation mechanisms attributable to latent, language-agnostic semantic representations. This transition features a notable divergence between layers responsible for maintaining copying (persisting into upper layers) and those increasingly favoring target-language outputs. Notably, the model's final WLT accuracy remains modest at 32.1%, but with a consistent increase for challenging pairs as translation mechanisms become more abstract.

Figure 2: Evolution of the translation-over-copy margin across model layers and checkpoints; copy-promoting layers are blue, translation-promoting layers are red.

Layer-Resolved Mechanistic Interpretability

Logit Lens Analysis

Application of the logit lens demonstrates that copy-promoting layer transitions (notably 7→8, 12→13, and 15→17) are rapidly established in the lower/mid transformer stack and persist across pretraining. However, upper layers graduately (but not globally) re-weight the target translation probability upwards, coinciding with the emergence of effective translation. Significantly, the translation-over-copy margin demonstrates that, while lower/intermediate layers promote copying, upper layers—especially in later checkpoints—incrementally privilege accurate translation, even where token overlap is minimal or absent.

Parameter-Level Influence (ExPLAIND)

By leveraging ExPLAIND for model/data attribution, the paper identifies bottom-layer value projections (0–2) as decisive in copy promotion, while middle-block parameters (6–9) become increasingly copy-suppressive in Phase II. Parameter upscaling in these blocks increases context/source copying, while downscaling dampens the copy error rate without substantially reducing translation accuracy. These findings provide evidence that early layers are responsible for both the induction and refinement of copying, while mid-layer parameters arbitrate the shift toward genuine translation.

Ablations and Blockwise Parameter Interventions

Layer-block swapping experiments further elucidate the division of labor across the transformer stack. Merging final-checkpoint intermediate blocks (layers 10–15) into earlier checkpoints causes WLT accuracy to rise by up to 30% (relative to the original at the same point), confirming that cross-lingual, language-agnostic semantic representations develop in intermediate layers by mid-pretraining. Contrastingly, mixing bottom and intermediate blocks yields suppression of spurious copy errors and further performance gains, indicating that both block classes are necessary for maximally selective cross-lingual mapping. Top-layer blocks alone are not sufficient for translation accuracy gains.

Figure 3: Accuracy trajectories from swapping learned parameter blocks into earlier checkpoints, highlighting the functional specialization developed throughout pretraining.

Error Decomposition and Role of Token Overlap

A detailed error taxonomy reveals that copying is more prevalent for language pairs with script overlap. The WLT dataset, by design, exposes that valid copies are rare (4.4%), while partial/zero-token overlap pairs constitute the majority of challenging translation cases. The combination of attribution analysis and output error distributions shows that suppression of copying—especially for non-overlapping pairs—is a central aspect of model improvement through pretraining, and is accomplished by coordinated transformations across lower and intermediate layer blocks.

Practical and Theoretical Implications

This work robustly demonstrates that cross-lingual generalization in decoder-only LLMs is not monolithic but entails a staged progression: token-level copying is rapidly induced and persists as a fallback, while abstract semantic parameterizations emerge in blockwise fashion and ultimately govern translation. These findings systematize and corroborate prior hypotheses about the encoding of "shared latent concept spaces" in multilingual transformers (cf. (Körner et al., 30 Jan 2026), [2025.acl-long.253]); the explicit checkpoint-level interventions provide compelling quantitative evidence of the functional specialization of transformer depths ([bandarkar-peng-2025-unreasonable], [koerner2026meaningsmeetinvestigatingemergence]).

Practically, these findings provide actionable guidance for model distillation, targeted fine-tuning, and layerwise pruning in multilingual settings. The evidence that generalizing mechanisms are largely in place by mid-pretraining suggests that curricula, checkpointing, and intervention can target this region for improved data efficiency or targeted enhancement, especially for low-resource or typologically divergent language pairs. The behavioral variability across language pairs, especially the asymmetry favoring translation into English, underscores the relevance of data balance and script design ([bagheri-nezhad-etal-2025-beyond], [limisiewicz-etal-2023-tokenization]).

Theoretically, this paper advances understanding of transfer and inductive biases in LLMs and grounds claims about shared representational spaces and the suppression of error modes through mechanistic, ablation-based evidence. The identification of specialized parameter blocks, their persistence, and their cross-lingual generality frames a precise architecture-centric account of how multilinguality arises, aligning with recent discovery of language- and domain-specific expert neuron activation ([liu-etal-2024-unraveling], [tang-etal-2024-language], [cao2024mindtonguesdeepdive]).

Future Directions

Future work should extend this blockwise/phase-resolved approach to contextual and reasoning tasks, examine scaling via larger models or corpora, and systematically manipulate data curricula and tokenization schemas. There is a critical need for direct probing of latent concept spaces via activation patching and neuron-level causal interventions ([dumas-etal-2025-separating], [tezuaka-inoue-2025-transfer]). Exploring how these mechanisms interact with instruction-tuned, post-hoc aligned, or supervised translation objectives will clarify the boundaries and limits of incidental versus explicit cross-lingual generalization.

Conclusion

This paper provides a high-resolution, mechanistic dissection of how multilingual transformer architectures move from surface copying strategies to genuine translation during pretraining. Through checkpoint-rich behavioral and interventionist analysis, it establishes that copying is both an early inductive bias and an error mode, subsequently reified and suppressed via development of cross-lingual latent spaces primarily in the model's intermediate layers. These results offer both explanatory and prescriptive insights for the science and engineering of multilingual LLMs.