COMMENTRA: Comment-Driven Code Translation

• COMMENTRA is a code translation paradigm that injects concise, purpose-driven natural language comments to maximize translation fidelity between programming languages.
• It employs a failure-driven, iterative two-stage process where comments are added only when initial LLM-based translations fail, significantly raising compilation and test pass rates.
• Empirical results demonstrate that targeted comment insertion can yield improvements of up to 205% in success rates, underscoring its practical impact on code translation benchmarks.

COMMENTRA designates a code translation paradigm that injects targeted, natural-language code comments into the translation workflow to maximize translation fidelity between programming languages. It was formally introduced and experimentally validated in "Revisiting the Role of Natural Language Code Comments in Code Translation" (Gupta et al., 23 Jan 2026), where it is shown to deliver significant and sometimes superlinear improvements in compilation and test-passing rates for LLM-based code translation frameworks. COMMENTRA operationalizes a selective, failure-driven comment augmentation procedure, grounded in large-scale ablations on comment type, intent, and placement.

1. Formal Problem Setting

Let $L_s$ and $L_t$ denote source and target programming languages, respectively. Given a code snippet $C \in L_s$ (e.g., a function or code block), and an optional associated natural-language comment block $M$, the objective is to produce a translation $T \in L_t$ such that $S(T) = 1$, where $S(\cdot)$ is a binary measure of translation quality: successful compilation and passage of all supplied unit tests. The translation function $f_\theta(\cdot)$ (LLM with parameters $\theta$) maps $(C, M) \mapsto T$. The optimal translation is $T^* = \arg\max_T P(S(T) = 1 \mid C, M)$. When $M = \emptyset$, the formulation recovers standard (comment-free) code translation.

2. COMMENTRA Motivation and Empirical Observations

COMMENTRA is motivated by empirical findings that code-specialized LLMs are exposed during pretraining to large amounts of commented code, causing their translation outputs to be sensitive to natural language guidance. Systematic experiments reveal multiple actionable insights:

• Targeted comment insertion can resolve failures in both syntax and logic, yielding up to +435% relative improvements in pass rates for some language pairs.
• Short, descriptive comments articulating overall code intent ("what this function does") provide more consistent guidance than multi-intent or excessively verbose comments.
• Line-by-line inline comments have the highest marginal benefit compared to method specifications or pseudocode.
• Indiscriminate or multi-intent comments can inject noise and reduce output quality.
• The maximal gains arise when comments are only injected upon encountering translation failures—a minimalist, cost-sensitive strategy.

Existing code translation benchmarks generally remove comments, thereby both masking the true capabilities of pretrained models and suppressing the potential benefits of comment-driven disambiguation.

3. Algorithmic Structure and Iterative Strategy

COMMENTRA employs an iterative, two-stage workflow:

1. Initial Translation Attempt: For each input $C_j$, produce $T_0 = f_{\rm translate}(C_j)$ without additional comments.
   • If $S(T_0) = 1$, record as success.
   • Otherwise, add $j$ to the set of failed cases.
2. Iterative Guided Translation: For up to $K$ iterations, and as long as failures remain:
   • For each failed $C_j$, generate a targeted comment $M_i = g_i(C_j)$ using a selected commenting LLM.
   • Retry translation: $T_i = f_{\rm translate}(C_j \| M_i)$.
   • Record successes and update the failure set.

Each new iteration targets only unresolved failures, avoiding comment injection for already-successful cases. No LLM fine-tuning or bespoke decoding is required; both translation and commenting LLMs operate in "out-of-the-box" mode with greedy decoding. The only infrastructure required beyond the LLMs is an automated test harness capable of compilation and unit test validation.

4. Experimental Landscape and Quantitative Results

The experimental campaign covers five languages (C, C++, Go, Java, Python) and twenty directed translation pairs, using the AVATAR (Java, Python) and CodeNet (C, C++, Go) benchmarks for a total of 1,100 unique, uncommented samples. Twenty models spanning five translation LLMs (CodeLlama-13B, DeepSeek-Coder-V2, GPT-4o-mini, Granite-8B-Code-Instruct, StarCoder-1) and three commenting LLMs (Mistral-7B, DeepSeek-Coder-V2, GPT-4o-mini) were evaluated.

Empirical results demonstrate that:

• Baseline (uncommented) translation achieves $A_0 \approx 30\%$ on average.
• One iteration with DeepSeek-coder comments recovers an additional $+6\%$ to $+51\%$ passing translations ($A_1$).
• A second iteration with GPT-4o-mini further increases the total gain, with maximum observed cumulative improvements exceeding $100\%$ (i.e., more than doubling the success rate).

Illustrative per-model cases:

• Granite-8B-Code-Instruct (Python$\to$Java): $A_0 = 51.6\%$, $A_1 = 60.4\%$ $(+17\%)$, $A_2 = 89.2\%$ $(+45\%)$.
• StarCoder-1 (Go$\to$Python): $A_0 = 14.8\%$, $A_2 = 45.2\%$ $(+205\%)$.

5. Ablative Analysis: Comment Intent and Placement

Two principal ablations clarify where COMMENTRA's gains originate:

• Intent analysis (CJBench): Automatically generated, single-intent comments provide $+25\%$–$+35\%$ accuracy gains, whereas author-written, multi-intent comments yield negligible or negative impact ($\pm5\%$).
• Placement analysis: Inline, line-by-line comments outperform method-level specs by $10$–$20\%$ and pseudocode by $15$–$30\%$ absolute.

These findings establish that the most substantial improvements derive from concise, purpose-focused, inline comments applied only upon initial translation failure.

6. Broader Implications and Limitations

COMMENTRA reframes code translation benchmarks to align better with real-world repositories, where code is heavily commented. Benchmarking without comments underestimates LLM capabilities and misleads evaluation. COMMENTRA's comment-injection procedure is LLM-agnostic and requires no retraining or parameter updates. Limitations identified include reliance on the quality of the commenting LLMs, the potential redundancy or interference effects from verbose or multi-intent comments, and the need for robust test harness support.

In sum, COMMENTRA provides a principled, empirically validated framework for leveraging natural-language comments as an adaptive resource for LLM-based code translation, triggering significant improvements in translation outcomes only when and where they are most needed (Gupta et al., 23 Jan 2026).