---
title: 'TransLibEval: Library Code Translation Benchmark'
url: https://www.emergentmind.com/topics/translibeval
type: topic
---

# TransLibEval: Library Code Translation Benchmark

TransLibEval is the first benchmark dedicated to library-centric code translation. It consists of 200 real-world tasks across Python, Java, and C++, each explicitly involving third-party libraries (TPLs) from diverse categories such as data processing, machine learning, and web development, with comprehensive dependency coverage and high-coverage test suites. The benchmark evaluates seven recent LLMs of commercial, general, and code-specialized families under six translation strategies of three categories—Direct, IR-guided, and Retrieval-augmented—and reports a dramatic performance drop compared with library-free settings, with average computational accuracy (CA) decline over 60% [2509.12087].

## 1. Motivation and problem setting

TransLibEval is motivated by the high prevalence of third-party library usage in practical software development. The benchmark description states that over 90% of codebases utilize third-party libraries, covering functions such as data processing, machine learning, and web development. It also identifies a benchmark gap: most prior code translation benchmarks are method- or class-level, with very limited library dependency coverage, described as “TPL-free” or $\leq 11\%$ coverage. This makes TPL-related translation errors hard to reveal and address, limiting progress in real-world migration scenarios [2509.12087].

Library-centric translation is presented as qualitatively harder than library-free translation. The stated challenges include correct identification of analog libraries, handling divergent APIs, type conventions, and semantics across programming languages. In this setting, translation quality depends not only on syntactic conversion, but also on whether the target program invokes useful libraries, preserves implementation intent, and remains executable under target-language dependency constraints.

A common misconception is that strong performance on library-free benchmarks implies readiness for practical code migration. TransLibEval directly contests that assumption by isolating tasks with explicit library dependencies and by exposing third-party reference errors that were previously obscured. This suggests that TPL handling is not a peripheral issue but a core determinant of translation robustness in realistic settings.

## 2. Benchmark design and construction

The benchmark is organized around four stated design principles: **Library-Intensive Tasks**, **Parallel Corpus**, **Comprehensive Testing**, and **Practical Relevance**. Concretely, it contains 200 real-world, method-level translation tasks. Python serves as the calibration language, and each task is translated from Python to Java and C++ by experienced engineers, following rigorous type and naming conventions [2509.12087].

Construction follows a two-phase process: Python reference version, then manual translation to Java and C++ following five strict principles—naming, implementation layout, type mapping, library selection, and test construction. The workflow includes expert reviewers, double-blind reviews, and final arbitration for complex or ambiguous translations. The resulting corpus is fully parallel across representative languages for controlled evaluation.

The benchmark covers 200 distinct TPLs across 10 major categories. The task distribution and representative libraries are as follows.

| Category | Sample libraries | #Tasks |
|---|---|---:|
| Data Processing | numpy, pandas; JDBI, Jackson; boost, Eigen | 68 |
| Utilities | tqdm; Guava, Commons IO; boost | 36 |
| Machine Learning | scikit-learn; deeplearning4j; xgboost | 28 |
| Web Development | requests; OkHttp; libcurl | 24 |
| Visualization | matplotlib; JFreeChart; plplot | 20 |
| NLP | nltk; Stanford CoreNLP; cryptopp | 20 |
| Graphics | opencv, PIL; Lucene; opencv | 8 |
| Database | peewee; JDBI; sqlite3 | 4 |
| Security | pycrypto; BouncyCastle; cryptopp | 4 |
| Other | jsonschema; Commons Lang | 4 |

The category design emphasizes explicit library dependence rather than incidental imports. This suggests that benchmark difficulty is concentrated in cross-language dependency mapping, API substitution, and type-consistent reconstruction rather than in bare algorithmic translation alone.

## 3. Coverage, testing, and relation to prior benchmarks

Each TransLibEval task has 5 unit tests using Python `unittest`, Java JUnit, or C++ GoogleTest, reaching 95% statement coverage on average. The test cases include normal, edge case, exception, type validation, and resource constraint scenarios [2509.12087]. This testing regime is intended to support reliable automatic assessment of compilation, execution, and functional correctness.

The benchmark is method-level and reports 41.0 LOC/Task. In the comparison excerpt, TransLibEval is contrasted with CodeNet, AVATAR, and PolyHumanEval on TPL coverage and testing.

| Benchmark | #TPLs (%) | Statement Coverage |
|---|---:|---|
| CodeNet | 21 (10.5%) | 1 per task |
| AVATAR | 51 (20.4%) | 25.1 per task |
| PolyHumanEval | 7 (4.3%) | 8.1 per task |
| TransLibEval | 200 (100%) | 5 per task (95%) |

Within this comparison, TransLibEval is distinguished by complete TPL involvement, larger per-task code size than the listed baselines, and high-coverage testing. A plausible implication is that it shifts benchmark emphasis from syntax-preserving translation toward dependency-aware, semantically executable migration.

## 4. Evaluation protocol, model set, and metrics

The evaluation studies seven LLMs: GPT-4o, GPT-3.5-Turbo, DeepSeek-V3, Qwen-Max, Llama3-8B, Llama3-70B, and CodeLlama. These are grouped as commercial, general-purpose, and code-specialized families. All experiments use standardized, deterministic sampling with `temperature=0` and `n=1` [2509.12087].

Four metrics are defined for translated output $\hat{y_k}$, ground truth $y_k$, and total samples $N$:

$$
CSR = \frac{\sum_{k=1}^N cs(\hat{y_k})}{N}
\quad \text{where} \quad
cs(\hat{y_k}) =
\begin{cases}
1 & \text{if compiles} \\
0 & \text{otherwise}
\end{cases}
$$

$$
PR = \frac{\sum_{k=1}^{N}\sum_{j=1}^{T_k} ps(\hat{y_k}, y_k, j)}{\sum_{k=1}^{N} T_k}
\quad \text{where} \quad
ps(\hat{y_k}, y_k, j) =
\begin{cases}
1 & \text{if test } j \text{ passes} \\
0 & \text{otherwise}
\end{cases}
$$

$$
CA = \frac{\sum_{k=1}^N ca(y_k, \hat{y_k})}{N}
\quad \text{where} \quad
ca(y_k, \hat{y_k}) =
\begin{cases}
1 & \text{if all tests pass} \\
0 & \text{otherwise}
\end{cases}
$$

$$
LDA = \frac{N_{lib}}{N}
$$

where $N_{lib}$ is the number of translated samples that correctly invoke useful libraries.

These metrics separate compilation viability, partial test success, full functional correctness, and library dependency awareness. The presence of LDA is especially notable: it operationalizes whether a translation uses appropriate library calls rather than merely producing compilable code.

## 5. Empirical findings on models, directions, and strategies

The headline result is a dramatic degradation relative to library-free settings. The benchmark reports an average drop of more than 60% in CA; an example given is CA dropping from 78%+ in TPL-free settings to 31–43% in TPL-involved settings [2509.12087]. This makes TransLibEval primarily a stress test for dependency-aware translation rather than a simple extension of existing benchmarks.

Under the Direct strategy, the reported results are as follows.

| Model | CSR | CA |
|---|---:|---:|
| GPT-4o | 75.42 | 42.67 |
| DeepSeek-V3 | 70.33 | 39.67 |
| Qwen-Max | 71.75 | 39.50 |
| GPT-3.5 | 56.75 | 25.92 |
| Llama3-70B | 51.17 | 28.25 |
| Llama3-8B | 39.08 | 19.92 |
| CodeLlama | 32.33 | 19.91 |

The corresponding PR values range from 21.76 for CodeLlama to 48.27 for GPT-4o. The benchmark states that commercial LLMs dominate, with up to 67.8% higher CSR and about 68% higher CA over smaller or code-specialized models. This directly counters the expectation that code-specialized models should necessarily lead in TPL-centric translation.

Translation direction matters substantially. Python as target performs best, with CA over 50%, while C++ and Java as target are much harder, with CA often below 18%. The stated explanation is a combination of pretraining bias, dynamic typing, and better library coverage for Python, versus strict typing and a sparser TPL ecosystem for Java and C++.

Strategy analysis is organized into three categories and six strategies: **Direct**; **IR-guided**, comprising **IR(CoT)**, **IR(pseudocode)**, and **IR(summary)**; and **Retrieval-augmented**, comprising **RA(method)** and **RA(name)**. The benchmark emphasizes that there is no universal best strategy. For C++ as target, **RA(name)** and **IR(pseudocode)** are reported as best; for Java as target, **RA(method)** and **Direct)** are best; and the strategy summary associates Python-target translation with IR-guided support. At the LDA level, the per-direction maxima are: **IR(pseudocode)** for C++$\rightarrow$Py at 71.43, **IR(summary)** for Ja$\rightarrow$Py at 76.19, **RA(name)** for Py$\rightarrow$C++ at 47.62 and Ja$\rightarrow$C++ at 52.38, and **Direct** for Py$\rightarrow$Ja and C++$\rightarrow$Ja at 66.67.

Library awareness shows a similar hierarchy. Commercial LLMs lead in identifying and applying correct TPLs, with DeepSeek reaching 60.2% LDA. By target language, translation to Python yields 57.9% LDA, while translation to C++ yields 37.3% LDA. This suggests that library selection remains especially difficult in less-supported ecosystems even when general code generation is strong.

## 6. Error taxonomy and research implications

The benchmark includes manual analysis of about 4,831 failed cases for GPT-4o alone. The dominant finding is that third-party reference errors account for an average 79.58% of all failures [2509.12087]. The top-level error distribution is summarized below.

| Error category | Share |
|---|---:|
| Third-Party Reference Errors | 79.58% |
| Generation Errors | 11.1% |
| Syntax Errors | 6.3% |
| Runtime Errors | 2.48% |
| Miscellaneous | 0.54% |

Within third-party reference errors, the benchmark distinguishes **library-related** errors (34.41%), **API-related** errors (50.1%), and **parameter-related** errors (15.49%). The library-related subgroup includes calling invalid library (36.44%), keeping original PL's library (44.44%), missing required libraries (11%), and missing import statements (8.12%). The API-related subgroup includes calling irrelevant APIs (55.31%), retaining original PL API (33.24%), missing APIs (5.79%), and calling non-existent API (5.66%). The parameter-related subgroup includes type mismatch (39.37%), return type errors (34.75%), and argument count errors (25.89%).

The benchmark also links failure modes to prompting strategies. **Direct** is most prone to “failed import” under A-1 because it often carries over libraries without adapting imports or library selection. **IR(summary)** has the highest API errors under A-2. **IR(CoT)** is prone to parameter errors under A-3 because the reasoning chain can drift. **RA(method)** has the fewest overall errors, while **IR(pseudocode)** and **RA(name)** are described as the best among IR-guided and retrieval-based strategies respectively, though still limited by abstraction loss or retrieval mismatches.

These results position library-centric translation as a major unsolved challenge. The benchmark’s own summary identifies four broader implications: library-centric translation remains a major unsolved challenge for LLMs; TransLibEval lays the foundation for rigorous, real-world code migration evaluation; strategy findings suggest retrieval augmentation and IR-guided prompting as focal directions, particularly for “TPL-poor” target languages; and the error case taxonomy reveals previously hidden TPL-centric vulnerabilities that future improvements and prompt or model tuning must specifically address.

Source: https://www.emergentmind.com/topics/translibeval