TRIFID: Translation Fidelity for Markdown

• TRIFID is a scoring framework that measures the structural fidelity of Markdown translations by evaluating code blocks, URL retention, and formatting accuracy.
• It uses three independent metrics—code, URL, and Markdown preservation—to provide clear diagnostic scores without conflating semantic accuracy.
• TRIFID supports translation-aware CI pipelines, enabling automated detection of structural regressions in multilingual open-source documentation.

TRIFID, expanded in the source as TriFidelity, is an early-stage translation fidelity scoring framework for Markdown-based technical documentation. It was introduced in the context of English-to-German translation of open-source README files, where LLMs can translate natural language accurately yet still corrupt code, hyperlinks, or Markdown structure. TRIFID therefore targets structural fidelity rather than semantic adequacy: it automatically checks how well a translation preserves code blocks and code content, URLs, and Markdown formatting, and is positioned as a first step toward translation-aware continuous integration pipelines for maintaining multilingual documentation (Adejumo et al., 4 Aug 2025).

1. Motivation and problem setting

README files in open-source repositories typically mix natural language text, code snippets, URLs or hyperlinks, and structural formatting such as headings, lists, tables, blockquotes, and emphasis. In that setting, a translation is only operationally useful if it preserves exact code, correct links, and the original structural layout. The motivating observation behind TRIFID is that LLM translations can remain linguistically acceptable while still dropping or duplicating code blocks, losing or altering hyperlinks, or changing Markdown structure (Adejumo et al., 4 Aug 2025).

This framing gives TRIFID a deliberately narrow scope. Its purpose is not to assess semantic correctness, fluency, or terminological adequacy. Instead, it provides a model-agnostic signal about whether the technical structure of a document has remained intact. The framework therefore answers the question “Did the translation keep the documentation’s technical structure intact?” rather than “Is the translation linguistically perfect?” (Adejumo et al., 4 Aug 2025).

The paper situates this need in a broader empirical context: translation activity in open-source repositories was found to be scarce, mostly in larger repositories and community-driven in nature. A plausible implication is that automation becomes especially attractive when multilingual documentation must be created or maintained at scale, but only if structural regressions can be detected automatically.

2. Pipeline architecture and scoring philosophy

TRIFID operates on a pair of Markdown documents: an original document $D_{\text{orig}}$ and a translated document $D_{\text{trans}}$. Its pipeline has three components: code block evaluation, URL evaluation, and Markdown formatting evaluation. Each component extracts the relevant structural elements from both documents, computes ratios or match scores, and maps those results to a normalized subscore in $[0,100]$ (Adejumo et al., 4 Aug 2025).

The framework does not define a single combined TRIFID score. Instead, it reports three separate scores—Code, URL, and Markdown—and then refers to averages across these as “preservation accuracy.” This design choice is significant because it preserves diagnostic resolution: a translation can perform well on formatting while failing on URLs, or preserve URLs while corrupting code. For technical documentation workflows, such separation is often more actionable than a collapsed scalar.

The scoring philosophy is intentionally operational. Code preservation rewards both the count of code blocks and exact content preservation after normalization. URL preservation rewards retention of original URLs and penalizes extra URLs introduced in translation. Markdown preservation treats structure as a category-wise count matching problem across seven element types. This suggests a framework optimized for automation and reproducibility rather than for nuanced human judgment.

3. Component metrics

Code preservation

TRIFID extracts code blocks from the original and translated documents and compares both their counts and their normalized contents. If $N_{\text{orig}}$ and $N_{\text{trans}}$ are the numbers of code blocks, the code block count ratio is defined as

$$N_{\text{orig}} = 0 \land N_{\text{trans}} = 0 \;\Rightarrow\; r_{\text{count}} = 1.0$$

and otherwise

$$r_{\text{count}} = \frac{\min(N_{\text{orig}}, N_{\text{trans}})} {\max(N_{\text{orig}}, N_{\text{trans}})}.$$

The count component contributes up to 50 points:

$$\text{CodeScore}_{\text{count}} = 50 \times r_{\text{count}}.$$

Content preservation is evaluated after normalization intended to ignore translated comments or example outputs. The normalization strips lines that are clearly comments or prompt/outputs, for example lines starting with #, $$</code>, or <code>*</code>. TRIFID then uses a strict binary criterion:</p> <p>$$\mathbb{1}(\text{content preserved}) = \begin{cases} 1, & \text{if all corresponding normalized blocks match} \ 0, & \text{otherwise.} \end{cases}$$</p> <p>The content component also contributes up to 50 points:</p> <p>$$\text{CodeScore}_{\text{content}} = 50 \times \mathbb{1}(\text{content preserved}).$$</p> <p>The total code score is therefore</p> <p>$$D_{\text{trans}}$0

A consequence of this formulation is that a single substantive code modification can reduce a translation from 100 to 50 even when code block counts match exactly (Adejumo et al., 4 Aug 2025).

URL preservation

For URLs, TRIFID extracts the distinct URL sets $D_{\text{trans}}$1 and $D_{\text{trans}}$2. It then defines

$D_{\text{trans}}$3

with

$D_{\text{trans}}$4

If $D_{\text{trans}}$5, the URL score is

$D_{\text{trans}}$6

If the original contains no URLs, TRIFID assigns

$D_{\text{trans}}$7

Scores are then rounded to the nearest integer in $D_{\text{trans}}$8. This metric gives 70% of the score to preserving original URLs and 30% to avoiding extra URLs. An important limitation of this first version is that if the original has no URLs, extra URLs introduced by the model are not penalized because the rule directly sets $D_{\text{trans}}$9 (Adejumo et al., 4 Aug 2025).

Markdown preservation

Markdown formatting is evaluated across seven categories: headers, bold, italic, unordered lists, numbered lists, tables, and blockquotes. Let $[0,100]$0 and $[0,100]$1 denote the counts for category $[0,100]$2, with $[0,100]$3. The per-category ratio is

$[0,100]$4

The overall Markdown score is the average of these category-wise ratios:

$[0,100]$5

This design makes Markdown preservation a count matching problem by category rather than a syntactic equivalence check. It captures structural loss or duplication, such as missing headings or altered emphasis counts, without attempting a full parse-level comparison (Adejumo et al., 4 Aug 2025).

4. Empirical evaluation

The reported evaluation used 50 README files from large-scale GitHub repositories with more than 10,000 stars or forks. The task was English-to-German translation, and the two models evaluated were OpenAI ChatGPT-4 (4.0) and Anthropic Claude 3.5 Sonnet. The prompt was:

“You are an expert translator, please translate the following Markdown content from English to German. Preserve code blocks, links and markdown format exactly as they are.” (Adejumo et al., 4 Aug 2025)

Across the 50 README translations, the average structural preservation scores were as follows.

Preservation	Claude Avg.	GPT-4 Avg.
Code Score	95.51%	85.24%
URL Score	93.66%	89.52%
Markdown Score	97.88%	92.80%

These results indicate that both models generally preserved structure well, with most averages above 85%, but Claude outperformed GPT-4 on all three structural metrics (Adejumo et al., 4 Aug 2025).

The paper also reports specific error patterns. Formatting inconsistencies affected 34% of translated README files for bold text, 32% for italic emphasis, and 22% for header structure. URL errors included dropped deep links, replacement with top-level homepage URLs, and extra references introduced by the model. Code issues included code block count mismatches and content modifications such as localized string literals that should have remained unchanged.

A second quantitative result is the negative association between README size and preservation accuracy. The reported Spearman correlations were:

Metric	Spearman $[0,100]$6
Claude_Code	-0.268958
Claude_Url	-0.610617
Claude_Markdown	-0.385345
GPT_Code	-0.435587
GPT_Url	-0.699682
GPT_Markdown	-0.499684

The strongest negative correlations were observed for URL preservation, especially for GPT-4. This suggests that long and structurally complex documentation creates particular difficulty for consistent link retention (Adejumo et al., 4 Aug 2025).

5. Role in translation-aware continuous integration

TRIFID was introduced as a first step toward translation-aware continuous integration pipelines. In that role, it functions as an automated structural gatekeeper for translated documentation. A typical workflow described for this use case begins when a pull request adds or updates translated Markdown files, or when an original README change triggers automatic re-translation. The CI job then locates original and translated documents, runs the evaluator, computes $[0,100]$7, $[0,100]$8, and $[0,100]$9, and compares them against project-specific thresholds (Adejumo et al., 4 Aug 2025).

The reconstruction gives an example policy requiring each metric to exceed 95. If a threshold is violated, the pipeline can report actionable diagnostics such as missing URLs, code block mismatches, or Markdown categories with mismatched counts. This structure makes TRIFID suitable both for pre-merge quality control and for iterative improvement loops in which contributors or automated systems correct detected structural regressions.

The same section also suggests a future feedback loop in which a translation model generates Markdown, TRIFID evaluates its fidelity, and low structural scores feed back into prompt optimization or fine-tuning. This suggests TRIFID could serve not only as an evaluation framework but also as a reward-like signal for translation systems specialized in technical documentation.

6. Limits, interpretation, and significance

TRIFID’s central conceptual boundary is that it does not judge semantic accuracy. A translation can therefore score highly on TRIFID while still mistranslating prose, provided that code, links, and formatting are preserved. Conversely, a semantically strong translation can score poorly if it alters Markdown structure or modifies executable code. This separation is intentional and reflects the framework’s emphasis on structure-aware evaluation (Adejumo et al., 4 Aug 2025).

Several limitations are explicit in the metric design. Code content preservation is coarse-grained and strict: it assigns full credit only if all corresponding normalized code blocks match, and otherwise assigns none for that half of the score. Markdown evaluation is count-based rather than syntax-aware, so it can capture structural imbalance without proving full Markdown equivalence. URL evaluation does not penalize hallucinated URLs when the original contains none, because $N_{\text{orig}}$0 in that case. No single aggregate score is defined, which preserves interpretability but leaves overall thresholding to downstream tooling.

Within those limits, TRIFID establishes a concrete operational niche. It formalizes structural fidelity for LLM translation of Markdown documentation, provides automatically computable metrics, and offers a basis for translation-aware CI in repositories where multilingual README maintenance would otherwise require costly manual inspection. The paper presents it as a foundation for automated LLM-driven support for creating and maintaining open-source documentation (Adejumo et al., 4 Aug 2025).