TransCoder-Test Benchmark

• TransCoder-Test Benchmark is a standardized evaluation suite for unsupervised program translation models that assess function-level translations between C++, Java, and Python.
• It comprises 948 algorithmic problem triplets with hand-written unit tests, enabling rigorous execution-based and exact-match evaluation across six language pairs.
• The benchmark’s zero-shot protocol and extended TransCoder-test-X version enhance translation reliability by focusing on compilation, functional correctness, and parameter harmonization.

The TransCoder-Test Benchmark is a central evaluation suite for unsupervised program translation systems, specifically designed to assess the capability of models translating between C++, Java, and Python at the function level. Developed in the context of the TransCoder project, it has become the de facto standard for execution-based evaluation of source-to-source translation models. The benchmark consists chiefly of triplets of aligned algorithmic problems, each implemented in all three target languages and furnished with small, hand-written unit test suites to assess functional correctness. Over time, the suite has been expanded and refined, notably in the form of TransCoder-test-X, which addresses certain intrinsic limitations of the original test suite. Its rigorous, zero-shot protocol—disallowing test data from contributing to model training—establishes a difficult, unbiased standard for real-world, unsupervised code translation.

1. Benchmark Genesis, Structure, and Language Coverage

TransCoder-Test originated with the work of Rozière et al. (NeurIPS 2020), who sought to benchmark unsupervised neural translation between C++, Java, and Python (Davis, 2020). The test set draws primarily on algorithmic coding challenges sourced from GeeksforGeeks. For each of 948 canonical problems, parallel function-level implementations are provided in all three languages, each governed by a standard interface and accompanied by approximately 10 unit tests. This “triplet” structure enables comprehensive cross-language translation evaluation in all six directed language pairs (e.g., C++→Java, Java→Python, etc.) (&&&1&&&, &&&2&&&). The problems span basic algorithms and data-structure manipulations, typically encoded as short, single function bodies (average 20–80 tokens), and restrict attention to core imperative constructs.

Corpus summary for original TransCoder-Test:

• Languages: C++, Java, Python 3
• Function count: ~948 per language (test split), with minor variance due to multi-line or formatting edge cases
• Average function size: Python 11.5 lines, Java 9.7 statements
• Test cases per problem: 10 (unit test harnesses)
• Domains: classic algorithms, elementary data structures, basic I/O

The suite is distributed as a pure testbin; no part enters model training or validation workflows.

2. Feature Coverage and Systematic Limitations

Manual audit of the TransCoder-Test corpus reveals substantial language restrictions, particularly for Java (Davis, 2020). The test functions exclude several core object-oriented constructs:

• No class or object definitions: The “class” keyword is absent from all test files.
• Absence of non-recursive user-defined function calls: All function invocations are either self-recursive or target the standard library.
• Unrepresented OOP features: No generics beyond parameterized library classes, no abstract classes or interfaces, absence of custom exceptions, no user-driven dynamic dispatch beyond built-in polymorphism.
• Omitted multi-file/project structure: Problems are single-function, with no imports, name resolution or build-system complexity.

In the subset of first 100 Java examples:

• 45 feature only elementary constructs (primitives, loops, arrays, basic I/O)
• 14 add Math library usage
• 2 permit recursion in addition to elementary
• 1 adds both recursion and Math calls
• 38 use “more sophisticated” features (switch/case, try/catch, library collections, wrapper classes), yet without any user-defined types or inter-procedural logic.

This restricted scope means that exact-match and functional results obtained on TransCoder-Test do not characterize the translation of industrial, multi-class object-oriented programs. The evaluation is best seen as quantifying model behavior on basic imperative and elementary library-using code.

3. Evaluation Protocols and Metrics

Three principal evaluation paradigms are used across papers leveraging TransCoder-Test and its successors (Davis, 2020, Huang et al., 2023, He et al., 30 Jan 2025):

1. Static, token-based metrics

• Exact-Match Accuracy (EM):

$$\mathrm{EM} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}(\mathrm{normalize}(\hat{y}_i)=\mathrm{normalize}(y_i))$$

where $N$ is the test set size, $\hat{y}_i$ is the predicted translation, $y_i$ is the reference, and $\mathrm{normalize}$ standardizes formatting and tokenization.

• BLEU Score:

$$\mathrm{BLEU} = \mathrm{BP}\;\exp\left(\sum_{n=1}^4 w_n\,\log p_n\right),\quad \mathrm{BP} = \exp\left(\min(1-\frac{L_\mathrm{ref}}{L_\mathrm{hyp}},0)\right)$$

Evaluates $n$-gram overlap between prediction and reference at the corpus level.

• CodeBLEU:

Applies syntax and data-flow heuristics to extend BLEU with Abstract Syntax Tree (AST) and data-flow-aware comparison. Employed in some recent studies but not in the original evaluation (He et al., 30 Jan 2025).

2. Unit-test-driven (execution-based) metrics

• Correct-Answer at Top N (CA@N):

$$\mathrm{CA}@N = \frac{1}{|D|}\sum_{i=1}^{|D|}\mathbf{1}\Bigl(\max_{1\le k\le N}\mathrm{TestPass}(y_{i,k}) = 1\Bigr)$$

For beam search over $N$ outputs per input, a translation is correct if any candidate passes all unit tests.

• TransCoder-test-X Additional Metrics (He et al., 30 Jan 2025):
  • Compilation Accuracy (CA): Fraction of outputs that compile.
  • Case Computational Accuracy (CCA): Fraction of test cases passed per problem (averaged).
  • Test Computational Accuracy (TCA): Fraction of problems for which all test cases pass.

Summary Table: Key Execution-Based Metrics (TransCoder-test-X)

Metric	Definition
CA	$\frac{1}{N}\sum_{i=1}^N \mathrm{compiled}(O_i)$
CCA	$\frac{1}{N}\sum_{i=1}^N \mathrm{pass}_i/T_i$
TCA	$$\frac{1}{N}\sum_{i=1}^N \mathbf{1}\{\mathrm{pass}_i=T_i\}$$

All compiled candidates are validated on the fixed input/output test set for their problem, with failure on any test case marking TCA=0 for that instance.

4. Published Model Results and Comparative Performance

Quantitative outcomes for major translation models on the benchmark highlight rapidly improving translation fidelity as models and protocols evolve:

TransCoder (original):

• Java→C++ EM: 90.2%
• C++→Java EM: 74.8%
• Java→Python EM: 68.7%
• BLEU (reported only in summary): Low 80s for Java→C++; mid 70s on other pairs

CoDist (Code Distillation):

• Achieves CA@1 (beam-10) on TransCoder-Test:
  • C++→Java: 82.1%
  • C++→Python: 67.9%
  • Java→C++: 87.9%
  • Java→Python: 68.1%
  • Python→C++: 86.9%
  • Python→Java: 81.1% (Huang et al., 2023)

ExeCoder (TransCoder-test-X):

• Enhanced test harness and metric suite (He et al., 30 Jan 2025):
  • CA: 92.68%
  • CCA: 87.69%
  • TCA: 83.04%
  • BLEU: 72.36%
  • CodeBLEU: 71.33%
  • Substantial improvements of 37–40 percentage points in execution-based metrics over CodeLLama; 1–2 percentage points over GPT-4o.

These results underscore the impact of richer execution-based evaluation and improved model architecture on practical translation reliability.

5. Identified Limitations and Successor Benchmarks

TransCoder-Test’s focus on short, stand-alone functions leaves several research gaps (Davis, 2020, He et al., 30 Jan 2025):

• Completely omits class/object-oriented design, custom types, and multi-file dependency resolution.
• Lacks non-recursive user- or system-defined inter-procedural calls.
• Narrow selection of data structure use—limited to primitive arrays and basic collection types with rigid interface conventions.
• Parameter mismatch between language versions and subtle test harness bugs that impede fair evaluation for functionally correct but structurally distinct solutions.

Proposed and realized remedies:

• TransCoder-test-X: Introduces parameter-passing and wrapper function augmentation, return type harmonization, and robustifies the compilation and execution pipeline (He et al., 30 Jan 2025).
• Suggested corpus extensions:
  • Test cases involving OOP hierarchies, generics, custom exceptions, multi-file/program structure, and mutable data structures.
  • Unit-test oracles providing canonical input/output pairs for semantic equivalence validation.

A plausible implication is that robust translation assessment at scale will increasingly require multi-function, multi-file benchmarks with compositional correctness, challenging current translation model paradigms.

6. Role in the Research Landscape and Ongoing Development

TransCoder-Test is the reference benchmark for zero-shot, execution-based evaluation of code translation models (Huang et al., 2023, He et al., 30 Jan 2025). Its execution-centric metric and unbiased, held-out design have led to widespread adoption. However, its focus on function-level, procedural challenges motivates ongoing efforts toward comprehensive program-level evaluation infrastructure.

Recent work has shifted toward:

• Enhancing test harnesses for broader implementation variants
• Expanding ground-truth and unit-test coverage
• Integrating structure- and semantics-aware similarity metrics (CodeBLEU, dynamic oracles)
• Tracking cross-language functional, syntactic, and compilation correctness under real-world library and build constraints

TransCoder-Test and its successors, such as TransCoder-test-X, continue to serve as essential benchmarks for the development and quantitative assessment of modern program translation systems, LLMs, and code distillation frameworks, facilitating reproducible, head-to-head comparison and progress tracking across the state of the art.