Evaluating In-Context Translation with Synchronous Context-Free Grammar Transduction

Abstract: Low-resource languages pose a challenge for machine translation with LLMs, which require large amounts of training data. One potential way to circumvent this data dependence is to rely on LLMs' ability to use in-context descriptions of languages, like textbooks and dictionaries. To do so, LLMs must be able to infer the link between the languages' grammatical descriptions and the sentences in question. Here we isolate this skill using a formal analogue of the task: string transduction based on a formal grammar provided in-context. We construct synchronous context-free grammars which define pairs of formal languages designed to model particular aspects of natural language grammar, morphology, and written representation. Using these grammars, we measure how well LLMs can translate sentences from one formal language into another when given both the grammar and the source-language sentence. We vary the size of the grammar, the lengths of the sentences, the syntactic and morphological properties of the languages, and their written script. We note three key findings. First, LLMs' translation accuracy decreases markedly as a function of grammar size and sentence length. Second, differences in morphology and written representation between the source and target languages can strongly diminish model performance. Third, we examine the types of errors committed by models and find they are most prone to recall the wrong words from the target language vocabulary, hallucinate new words, or leave source-language words untranslated.

• The paper introduces a novel method using SCFGs to rigorously evaluate in-context machine translation without data contamination.
• It quantitatively assesses the impact of grammar size, sentence length, morphology, and orthography on translation accuracy via exact-match metrics.
• It highlights that while LLMs accurately handle syntactic reordering, challenges persist in token-level translation with unfamiliar scripts and inflectional agreements.

Evaluating In-Context Translation with Synchronous Context-Free Grammar Transduction

Motivation and Background

The paper "Evaluating In-Context Translation with Synchronous Context-Free Grammar Transduction" (2604.07320) systematically investigates the capacity of LLMs to perform in-context machine translation (ICMT) using grammatical descriptions, in lieu of extensive parallel corpora. The study leverages synchronous context-free grammars (SCFGs) to generate formal, controllable analogues to natural language translation tasks. This approach isolates the contribution of explicit in-context grammar and vocabulary information, thus minimizing confounds from prior exposure or memorization during pretraining.

The work is situated within the ongoing debate on whether LLMs genuinely utilize in-context grammatical descriptions or defer to example-based reasoning, as suggested in prior evaluations with low-resource languages [tanzer-2023-benchmark, aycock-2024-LLMs]. By using formally generated languages with guaranteed novelty and SCFGs that support rigorous, exact comparison, the paper provides a more principled evaluation of the symbolic and combinatorial reasoning underpinning ICMT.

Experimental Design and SCFG Formalization

The core methodology employs parameterized SCFGs to instantiate pairs of context-free languages, explicitly modeling critical structural properties: grammar size, sentence length, word order, inflectional morphology, and orthographic conventions. Each sample includes the grammar, a source-language sentence, and an evaluation of the model's translation.

This setup offers three key advantages:

• No data contamination: All vocabularies are novel, preventing leakage from pretraining;
• Rigorous evaluation: Exact-match metrics are available, circumventing the inadequacies of heuristic overlap-based metrics (BLEU, chrF++);
• Controlled ablation: The effect of individual linguistic factors can be quantitatively measured, unattainable in real-world low-resource language tasks.

Key Findings and Quantitative Results

Grammar Complexity and Sentence Length

Translation accuracy is highly sensitive to both grammar size and sentence length. GPT-5 achieves near-perfect exact-match accuracy with small grammars (<100 rules) and short sentences (<10 words), but performance degrades sharply for grammars approaching natural-language scale (several thousand rules) and for longer strings (20+ words), with a drop from >95% to <50% accuracy.

Figure 1: Mean accuracy as a function of grammar size, sentence length, morphology, and orthography, highlighting major bottlenecks in translation performance.

Figure 2: Exact-match accuracy as grammar size and sentence length increase across models, demonstrating model-dependent degradation.

Word Order

The impact of target-language word order (SVO, SOV, OVS) is negligible for GPT-5 and comparable LLMs. Models consistently preserve structural reordering as specified in the SCFG, indicating robust attention to supplied syntactic rules over training-induced prior biases.

Figure 3: GPT-5 performance on word order variations; error bars show insensitivity to word order modulations.

Morphology

The presence of inflectional agreement (person, number) in the target language reduces accuracy substantially, with lowest performance observed when transforming from a non-agreeing to an agreeing language (NoAgr → Agr). The challenge arises because the mapping becomes one-to-many, requiring symbolic alignment with grammatical features rather than token-level translation.

Figure 4: Detailed model performance on morphological agreement settings, quantifying the translation difficulty across morphological paradigms.

Orthography

Translation into less frequent or unfamiliar scripts (Cyrillic, Hebrew, Hebrew with vowel points) leads to pronounced accuracy drops and outright failure in fully pointed Hebrew. The effect is orthogonal to syntactic and morphological complexity, reflecting token-level biases induced by training corpora frequency. Heuristic metrics (BLEU, chrF++) consistently overestimate performance, especially as script unfamiliarity rises.

Figure 5: Full accuracy breakdown for GPT-5 across target orthographies, revealing script-specific collapse.

Figure 6: Gemma-3-12b-it performance for orthographic variants; Latin script is markedly easier than Cyrillic or Hebrew.

Error Taxonomy

A multi-class error analysis identifies:

• Source vocabulary leakage: Direct copying of source words into output, violating grammar constraints;
• Recall errors: Substituting target vocabulary words incorrectly;
• Word omissions: Missing target words, especially in agreement-heavy outputs;
• Orthographic errors: Producing forms in the wrong script or hallucinating non-grammar words.

  Figure 7: Distribution of major error types by experiment for GPT-5, quantifying prevalence for output hallucination, recall, copying, and omission.

Implications for LLM Translation and Future Directions

The findings emphasize that current LLMs can utilize in-context formal language descriptions for symbolic translation, but their robustness is fundamentally constrained by grammar size, input length, morphology, and orthographic novelty. The negligible influence of word order establishes that syntactic reordering is not a limiting factor; rather, the scale and richness of grammatical inventory are decisive.

Orthographic brittleness, especially in unfamiliar scripts, suggests that token-level generation is strongly conditioned by frequency statistics from pretraining corpora, limiting generalization to new language forms. Exact-match evaluation highlights systematic inflation by standard overlap metrics, especially as complexity or novelty increases, necessitating more rigorous metrics in practical evaluation.

Theoretically, this work supports a dual-bottleneck model for ICMT: symbolic rule induction and token-level robustness for unfamiliar representations. Practically, it underscores that expansion to truly low-resource languages will require advances in symbolic reasoning and robust handling of novel scripts.

Future development in AI translation systems will need to address these bottlenecks, likely by augmenting grammar-informed symbolic reasoning, enhancing robustness to unfamiliar orthographies, and integrating mechanisms for reliable cross-linguistic morphological mapping. A hybrid approach leveraging few-shot examples alongside grammatical descriptions may mitigate current shortcomings.

Conclusion

This paper provides a formal, principled and quantitative evaluation of ICMT via SCFG transduction, revealing that LLMs' ability to translate using in-context grammars is real, but strongly limited by complexity and orthographic novelty. The work establishes methodological standards for isolating symbolic reasoning in translation and identifies critical directions for overcoming representational bottlenecks in general-purpose translation systems.