Fabricator or dynamic translator?

Abstract: LLMs are proving to be adept at machine translation although due to their generative nature they may at times overgenerate in various ways. These overgenerations are different from the neurobabble seen in NMT and range from LLM self-explanations, to risky confabulations, to appropriate explanations, where the LLM is able to act as a human translator would, enabling greater comprehension for the target audience. Detecting and determining the exact nature of the overgenerations is a challenging task. We detail different strategies we have explored for our work in a commercial setting, and present our results.

• The paper presents a refined taxonomy categorizing LLM overgeneration into oscillatory, detached, and minimally detached types.
• The study develops complementary detection strategies using an MTQE model and alignment-based CheckAlign to capture both obvious and subtle translation errors.
• Results highlight the challenge of distinguishing beneficial explicitation from harmful fabrications, suggesting the need for human-in-the-loop refinements and advanced methods.

Distinctive Overgenerations in LLM Machine Translation: Detection and Taxonomy

Introduction

The paper "Fabricator or dynamic translator?" (2604.15165) investigates the generative behavior of LLMs in Machine Translation (MT), focusing notably on the phenomenon of overgeneration—output that is fluent and contextually plausible, but not strictly faithful to the source input. Unlike the classical neurobabble and hallucinations observed in Neural Machine Translation (NMT), overgenerations in LLM MT are more nuanced and encompass fabricated content, explicitation, and self-explanations. This work constructs a taxonomy of these phenomena, explores the lack of annotated datasets in commercial LLM MT, and proposes robust detection strategies targeting various overgeneration types.

Taxonomy of Overgenerations

The authors introduce a refined taxonomy, expanding beyond the oscillatory and detached hallucinations typical in NMT. Notably, the paper differentiates:

• Oscillatory: Repetitive, neurobabble-like output, readily detectable.
• Detached: Fluent, but largely ungrounded in the source; includes refusals, self-explanations.
• Partially Detached: Contains coherent chunks not present in the source, such as translation context explanations or prefixes.
• Minimally Detached: Subtle, seamless additions—single words or short phrases—that are difficult to distinguish from correct expansions or natural target renderings.

This granularity is critical, as minimally detached confabulations are highly challenging to pinpoint due to the high semantic and fluency overlap with well-formed, contextually explicit translations.

Dataset Construction

A notable contribution is the construction and curation of diverse datasets reflecting real-world translation settings, including synthetic, open-source, and internally annotated test sets. Key sources include the WMT25 AOC task, DeepSpin dataset, R&D and APE internal sets, and custom minimally detached collections. The scarcity of naturally occurring, labelled overgeneration examples in LLM MT output motivates the reliance on manual annotation and synthetic data augmentation.

Detection Strategies

Two complementary strategies are developed to detect overgeneration:

MTQE Model

An in-house Multilingual MT Quality Estimation (MTQE) model based on XLM-R Large is fine-tuned for overgeneration detection. Enhancement through synthetic overgeneration examples allows direct penalization of such content:

• MTQE delivers high precision (up to 1.00) for oscillatory and detached categories but suffers low recall on minimally detached confabulations due to their nuanced nature.

CheckAlign (Alignment-Based Detection)

Leveraging cross-lingual alignment models (AwesomeAlign, fine-tuned for en-it), the CheckAlign approach flags unaligned target chunks (threshold $n=2$). This strategy is suited for detecting detached and minimally detached output, especially where MTQE recall is insufficient:

• The alignment-based granularity enables recall of 0.77 on minimally detached test sets, but precision is compromised (0.22), primarily due to high false positives from legitimate explicitation or imperfect alignments.

Ensemble Approach

Combining MTQE and CheckAlign provides improved recall and robustness. The ensemble exploits distinct strengths: precise detection of overt errors (MTQE) and sensitivity to subtle, contextually plausible additions (CheckAlign).

Empirical Results

On open and internal test sets, the detection methods exhibit differentiated performance:

• Oscillatory/Detached: MTQE achieves nearly perfect precision and competitive recall (e.g., 0.86/0.80/0.83 F1 for WMT24-AOC).
• Minimally Detached: CheckAlign reaches recall of 0.77, but precision is significantly lower (0.22), indicating the challenge of distinguishing explicitation from genuine confabulation.

  Figure 1: Confusion matrix showing the effectiveness and limitations of CheckAlign on minimally detached test sets; false positives stem from explicitation and alignment artifacts.

Analysis reveals that many false positives correspond not to incorrect output but to legitimate expansions or explicitation—translation strategies favored by human professionals for target audience clarity. Examples include expansion of acronyms ("NSW" → "New South Wales") and insertion of implicit contextual nouns ("Al’s" → "Al’s shop"), reflecting translator intervention rather than model confabulation.

Practical and Theoretical Implications

This paper underscores a fundamental shift in MT error typology with LLMs. The transition to generative models introduces finely grained overgeneration behaviors that blur the boundaries between pathological hallucination and desirable explicitation. Consequently, the challenge in MT evaluation and quality assurance moves from detection of overt errors to differentiation between harmful fabrications and beneficial contextual adaptations.

Detection pipelines integrating MTQE and alignment-based heuristics are necessary but not sufficient; substantial human-in-the-loop refinement and further modeling are needed to resolve ambiguity at the explicitation-confabulation frontier.

Speculation on Future AI Developments

The high prevalence of nuanced overgenerations implies that future AI translation systems must support genre-sensitive, audience-adaptive criteria, not just literal fidelity. Detection systems must internalize explicit knowledge of translation norms, pragmatics, and contextual adaptation. Robust annotation frameworks and large-scale datasets capturing explicitation vs. confabulation distinctions will enable more sophisticated evaluation and control of LLM output.

Moreover, ongoing research should address cross-lingual alignment imperfections, especially for low-resource languages and complex language pairs. Improvement in unsupervised plug-in detectors—perhaps leveraging optimal transport and token-to-token interaction measures—holds promise for reliable automation in industrial translation workflows.

Conclusion

The study establishes a compelling new taxonomy and detection pipeline for LLM-induced overgeneration in MT, highlighting both challenges and avenues for progress. While fluent minimal additions remain hard to detect automatically, alignment-based and MTQE approaches together offer a practical compromise. The tension between confabulation and explicitation is a critical issue for translation quality, necessitating sophisticated, context-aware detection and labeling architectures. Future research must focus on annotation granularity, genre specification, and pragmatic translation norms to fully harness the dynamic adaptability of LLMs while guarding against risky fabrication.