PARATRANS: HPC Code Translation Benchmark

• PARATRANS is a cross-paradigm code translation benchmark for HPC that aligns Serial, OpenMP, and CUDA implementations to ensure executable correctness.
• The benchmark is curated from HeCBench and uses operational metrics like compilation and validation rates to assess translation quality across various directions.
• Experimental evaluations reveal that repair-based methods and latent reasoning guidance significantly improve both syntax and functional correctness in parallel-code translations.

PARATRANS, spelled “ParaTrans” in the paper body, is a cross-paradigm code translation corpus and benchmark for high-performance computing (HPC) introduced within UniPar to evaluate translation among Serial code, OpenMP, and CUDA. It was created because the authors argued that there was no standardized dataset for translation between parallel programming paradigms. In subsequent work, the same benchmark was used not only for zero-shot, few-shot, fine-tuned, and repair-based large-language-model evaluation, but also as a held-out behavioral benchmark for test-time latent reasoning guidance in executable parallel-code translation (Bitan et al., 15 Sep 2025, Bitan et al., 3 Jun 2026).

1. Definition and task scope

PARATRANS is organized around aligned implementations of the same benchmark kernel in different programming paradigms. In the reported experiments, the supported paradigms are Serial, OpenMP, and CUDA, and the benchmark is instantiated through four translation directions: Serial$\rightarrow$OpenMP, Serial$\rightarrow$CUDA, CUDA$\rightarrow$OpenMP, and OpenMP$\rightarrow$CUDA. Although the dataset construction yields aligned tuples that enable translation between any pair of languages among Serial, OpenMP, and CUDA, the published train/test split and evaluation focus on those four directions (Bitan et al., 15 Sep 2025).

The benchmark is not defined through textual similarity to a reference translation. Its central objective is executable correctness under paradigm change: code must be translated into a different parallel programming model while preserving behavior and satisfying the target API’s requirements. This emphasis makes PARATRANS distinct from ordinary code-to-code generation corpora, because the relevant failure modes are often deferred until compilation, execution, or semantic validation. In later work on latent reasoning guidance, this delayed feedback is treated as a core property of the benchmark rather than a nuisance, because parallel translations require nontrivial choices about parallel structure, synchronization, memory movement, and API-specific syntax (Bitan et al., 3 Jun 2026).

2. Corpus construction and benchmark composition

PARATRANS is built from HeCBench, which contains 499 benchmarks and 1758 benchmark implementations across several paradigms, including CUDA, HIP, SYCL/DPC++, and OpenMP-4.5. The curation process begins by retaining only benchmarks implemented in at least one of CUDA or OpenMP. The authors then isolate the kernel-centric source unit, remove comments, and create serial kernels from OpenMP implementations by removing OpenMP pragmas. Quality control is automatic: each code sample must compile successfully and produce outputs consistent with expected ground-truth results. Three benchmark kernels did not pass validation and were removed. The corpus is then pruned by token length using the LLaMA-3.3-70B-Instruct tokenizer, with a 7,500-token cutoff; after pruning, 380 CUDA-based programs and 258 OpenMP-based programs remain (Bitan et al., 15 Sep 2025).

The resulting benchmark is split with approximately a 9:1 train:test ratio. The reported pair counts are as follows.

Direction	Train	Test
Serial $\rightarrow$ OpenMP	235	20
Serial $\rightarrow$ CUDA	221	19
CUDA $\rightarrow$ OpenMP	221	18
OpenMP $\rightarrow$ CUDA	221	19

The total size is therefore 898 training pairs and 76 test pairs. The domain coverage inherits HeCBench’s breadth, including cryptography, machine learning, signal processing, and other categories. The authors also note a construction asymmetry that materially affects task difficulty: the Serial$\rightarrow$OpenMP direction is structurally easier because the serial source is produced by removing pragmas from OpenMP code, making source and target unusually similar. By contrast, Serial$\rightarrow$CUDA and the OpenMP/CUDA cross-paradigm directions require introduction or remapping of GPU execution structure, APIs, and memory behavior (Bitan et al., 15 Sep 2025).

3. Evaluation semantics and benchmark protocol

PARATRANS is evaluated with execution-based metrics rather than reference-text metrics. In the UniPar study, the two reported metrics are compilation rate, defined as the percentage of generated programs that compile successfully, and validation rate, defined as the percentage of generated programs that compile, execute, and produce outputs matching expected benchmark outputs. The paper does not provide symbolic formulas for these metrics; they are defined operationally. For functional checking, the generated program’s main function is replaced with the ground-truth kernel’s main, because the ground-truth main contains the verification logic. If this replacement causes compilation failure although the generated code previously compiled, the evaluation allows three GPT-4o-mini repair attempts while checking that kernel functionality is not altered (Bitan et al., 15 Sep 2025).

This behavioral orientation becomes even more explicit in the latent-guidance study. There, a translation on ParaTrans is counted as successful only if it compiles, runs, and passes integrated validation against the reference behavior. The benchmark’s 76-task held-out test set is split across CUDA$\rightarrow$0OpenMP (18), OpenMP$\rightarrow$1CUDA (19), Serial$\rightarrow$2OpenMP (20), and Serial$\rightarrow$3CUDA (19). The benchmark is therefore not merely a collection of aligned code pairs; it is a constrained executable translation testbed in which textual plausibility is insufficient (Bitan et al., 3 Jun 2026).

A plausible implication is that PARATRANS occupies an intermediate position between code-generation benchmarks and systems benchmarks. It is small enough to support controlled model comparison, but its pass criterion is behavioral and delayed, which makes it unusually suitable for studying repair loops, reranking, and pre-decoding guidance.

4. UniPar experiments and benchmark behavior

Within UniPar, PARATRANS is used for four major LLM usage modes: hyperparameter optimization for decoding, zero- and few-shot prompting, supervised fine-tuning, and iterative feedback through compiler-based repair. The evaluated base models are GPT-4o-mini and LLaMA-3.3-70B-Instruct. The abstract reports that off-the-shelf models struggle under default settings, with GPT-4o-mini reaching 46% compilation and 15% functional correctness, while the combined UniPar methodology improves performance by up to about $\rightarrow$4, reaching 69% compilation and 33% correctness (Bitan et al., 15 Sep 2025).

The detailed results expose several benchmark-specific regularities. First, compilation is much easier than validation: base LLaMA without an agent averages 0.49 compilation but only 0.1324 validation, and base GPT averages 0.46 compilation but 0.152 validation. Second, standard few-shot prompting is largely ineffective. For LLaMA, average compilation changes from 0.49 in 0-shot to 0.43 in 1-shot, 0.48 in 2-shot, and 0.29 in 3-shot; for GPT, it changes from 0.46 to 0.415, 0.45, and 0.42. Third, supervised fine-tuning improves semantic correctness more than raw compilability. For LLaMA, average validation rises from about 0.132 to 0.34, while average compilation rises only from 0.495 to 0.528. Fourth, agentic repair substantially improves both syntax and correctness. Base LLaMA validation increases from 0.1324 to 0.327 with the repair agents, and base GPT increases from 0.152 to 0.334. The best reported overall validation rate is 39.7% for fine-tuned LLaMA with agentic repair (Bitan et al., 15 Sep 2025).

Direction-wise, Serial$\rightarrow$5OpenMP is consistently the easiest task, and the hardest directions are generally Serial$\rightarrow$6CUDA and OpenMP$\rightarrow$7CUDA. The authors attribute the ease of Serial$\rightarrow$8OpenMP directly to corpus construction: the serial version is derived from OpenMP by removing pragmas. This suggests that PARATRANS is not a uniform difficulty benchmark; instead, it contains both near-surface structural rewrites and genuinely cross-model parallelization problems.

5. ParaTrans as a benchmark for latent reasoning guidance

The later paper “Latent Reasoning Guidance for Parallel Code Translation” repurposes ParaTrans as a benchmark for pre-decoding latent branch selection. The primary generator is a frozen LLaMA-3.3-70B model run in a COCONUT-style latent reasoning mode, while the guidance model is a Qwen-Coder-7B Process Reward Model (PRM) with a scalar value head and an 8192$\rightarrow$93584 adapter. At inference, the system performs 12 latent steps and, at each step, evaluates 8 candidate hidden-state branches—the original state plus 7 perturbed versions—before decoding the final code once. The central claim is that PARATRANS is an effective setting for this intervention because correctness is behavioral and delayed, so early hidden-state decisions about parallel structure can have large downstream effects (Bitan et al., 3 Jun 2026).

On the 76-task test set, the three-run mean executable validation rate without repair is 14.55 $\rightarrow$0 2.55 for vanilla LLaMA-3.3-70B, 33.48 $\rightarrow$1 1.11 for the fine-tuned LLaMA-3.3-70B baseline from UniPar, 32.89 $\rightarrow$2 3.44 for unguided latent reasoning, 27.28 $\rightarrow$3 6.34 for random latent branch selection, and 42.10 $\rightarrow$4 2.28 for latent PRM guidance. The improvement from 32.89% to 42.10% is 9.21 percentage points, with a paired task-bootstrap 95% CI of $\rightarrow$5 and a two-sided paired sign-flip $\rightarrow$6. Under a three-attempt repair loop, the mean validation rates are 30.67 $\rightarrow$7 4.34 for vanilla, 36.00 $\rightarrow$8 3.11 for fine-tuned, 36.40 $\rightarrow$9 5.47 for latent reasoning, and 45.18 $\rightarrow$0 3.31 for latent PRM guidance, yielding an 8.78-point gain over unguided latent reasoning with 95% CI $\rightarrow$1 and $\rightarrow$2 (Bitan et al., 3 Jun 2026).

Several auxiliary findings reinforce the benchmark’s suitability for this kind of study. A search-budget ablation shows 28.94% validation for $\rightarrow$3, 35.52% for $\rightarrow$4, and 42.10% for $\rightarrow$5. On the 952-sample PRM training set, 95% of trajectory samples had at least one candidate branch with higher terminal reward than the original continuation, 59% of candidate continuations improved over the corresponding unperturbed branch, and among samples with at least one trajectory that passes both validation and output comparison, 25% succeeded only through branching. A post-decoding control is also informative: a text PRM over 8 complete outputs reaches 25.00% validation, while oracle@8 over the same candidate sets reaches 36.84%, both below latent PRM guidance at 42.10%. The paper therefore treats ParaTrans not merely as a benchmark of translation accuracy, but as evidence that executable parallel-code translation contains useful nearby latent continuations that can be exploited before any final code is emitted (Bitan et al., 3 Jun 2026).

6. Validity conditions, limitations, and current interpretation

PARATRANS has several strengths as a benchmark. It supplies aligned kernel pairs across multiple parallel paradigms, evaluates executable behavior rather than textual similarity, and has been reused across distinct methodological settings, from fine-tuning and repair to latent hidden-state guidance. The latent-guidance paper also reports a leakage audit: PRM dev = 60, final PRM train = 952, branch-selection PRM train = 832, branch-selection source trajectories = 120, and ParaTrans test = 76, with the explicit statement that the 76-task test set is held out from all PRM training, development, trajectory collection, reward construction, and hyperparameter selection, and that there is zero overlap even when kernel names are compared ignoring translation direction (Bitan et al., 3 Jun 2026).

At the same time, the benchmark has clear limitations. The held-out test set contains only 76 pairs. The Serial$\rightarrow$6OpenMP direction is partly synthetic because serial instances are produced by stripping OpenMP pragmas, which makes that direction easier than genuinely independent automatic parallelization. The UniPar evaluation checks syntax and functional correctness, but not runtime performance, scalability, or quality of parallelization from an HPC optimization perspective. In the latent-guidance study, the authors make a deliberately bounded claim: the evidence comes from one base-model family, one PRM backbone, and one benchmark, so the results should be read as evidence for this specific setting rather than as a universal conclusion about latent guidance. They also note that the method requires access to intermediate hidden states and costly task-specific supervision, since each labeled example needs multiple rollouts and executable or semantic validation (Bitan et al., 15 Sep 2025, Bitan et al., 3 Jun 2026).

This suggests that PARATRANS is best understood as a compact, behaviorally grounded benchmark for executable parallel-code translation rather than a complete evaluation framework for HPC code migration. Its main research value lies in exposing the gap between compilability and functional correctness, and in providing a controlled setting where pre-decoding guidance, supervised adaptation, and repair-based test-time computation can be compared under a common executable criterion.