Kaizen: Metamorphic Fuzzing and Differential Testing for LLM-Translated HPC Applications

Abstract: LLMs are increasingly used to port scientific codes across heterogeneous high-performance computing (HPC) programming models, such as translating CUDA to OpenMP, OpenACC, Kokkos or SYCL. However, current evaluations use compilation success, token-level similarity, or developer-written tests from static benchmarks, which cannot reliably ensure behavioral correctness. We present Kaizen, a metamorphic fuzzing and differential testing framework for evaluating the correctness of LLM-translated HPC code. Kaizen uses metamorphic fuzzing via source-code mutation to generate semantically equivalent programs, grammar-based input fuzzing to explore behavioral diversity, and differential testing to expose semantic divergences between original and translated applications that compile and pass developer-written tests yet produce incorrect scientific results. We evaluate Kaizen on CUDA-to-OpenMP translation of 16 scientific applications from seven domains using three fine-tuned LLMs at kernel-level and full-program granularity. Our evaluation reveals that (1) compilation success is a poor proxy for correctness; (2) LLM-translated programs exhibit systematic compile-time error patterns, with nine categories for kernel-level translation and 27 for full-program translation; (3) semantic errors that survive compilation are often input-dependent and require differential testing to expose; and (4) full-program translation is substantially harder than kernel-level translation. These findings highlight the need for correctness-oriented evaluation of LLM-assisted HPC code translations.

• The paper presents Kaizen, a framework that uses metamorphic fuzzing and differential testing to rigorously evaluate the semantic correctness of LLM-translated HPC codes.
• It leverages 15 grammar-based mutation operators and runtime input fuzzing to systematically reveal hidden errors that standard compilation checks miss.
• Experimental results show that high compilability does not ensure correctness, with kernel-level translations achieving up to 72% correctness while full-program translations remain challenging.

Kaizen: Correctness-Oriented Evaluation of LLM-Based HPC Code Translation

Introduction and Motivation

The emergence of LLMs as code translators for scientific and HPC domains—particularly for translating CUDA code to performance-portable models like OpenMP, SYCL, or Kokkos—promises to accelerate porting legacy applications to modern heterogeneous hardware. However, standard evaluation protocols, based on shallow proxies such as compilation success rates, token-level similarity, or sparse functional test suites, do not reliably ensure semantic preservation in scientific codes where behavioral correctness is paramount. The "Kaizen" framework (2607.04058) addresses this gap by introducing a rigorous methodology based on metamorphic fuzzing, grammar-based input fuzzing, and systematic differential testing. Its goal is to uncover semantic divergences in LLM-translated HPC codes, even in cases where naive metrics would declare the translation successful.

Methodology: The Kaizen Framework

Kaizen is architected as a multi-stage pipeline designed for thorough, adversarial evaluation of LLM-based code translators. It introduces semantic diversity both at the source code and runtime input levels to forcibly exercise translation robustness.

Kaizen operates in parallel along two translation granularities:

• Kernel-level translation, in which individual compute kernels are extracted and translated.
• Full-program translation, where the entire application code (including host orchestration, memory management, data movement, and device code) is translated en bloc.

Figure 1: Kaizen framework architecture showing parallel kernel-level and program-level translation evaluation paths, both leveraging source code and input fuzzing for differential semantic validation.

Metamorphic Source-Code Fuzzing

Kaizen employs 15 grammar-based mutation operators to produce large families of semantically equivalent source variants. These operators inject dead code, permute variable orders, manipulate comments, and perform identifier renaming, all without affecting observable semantics given developer-supplied inputs. Unlike random mutation approaches, this grammar-aware strategy ensures variants remain well-formed and do not introduce spurious faults unrelated to translation. Importantly, this component mitigates training data memorization in LLMs, differentiating true translation ability from simple regurgitation of memorized code.

Grammar-Based Runtime Fuzzing and Differential Testing

For each translation, Kaizen generates diverse input configurations through runtime grammar-based fuzzing, systematically varying array sizes, parameter values, data patterns, and boundary cases. Original and translated binaries are run in parallel on the same input corpus, and outputs are compared using configurable numeric norms to distinguish between benign and substantial divergences, accounting for floating-point inexactness where necessary. The approach flags discrepancies as correctness violations.

Experimental Evaluation

Evaluation focused on CUDA-to-OpenMP translation across 16 canonical scientific workloads from the HeCBench suite, spanning computational physics, geospatial analytics, computer vision, deep learning, and neuromorphic models. Three fine-tuned LLM models (ChatPORT-CL_13B, ChatPORT-SCB_7B, ChatPORT-HPC_C_6.7B) were assessed at both kernel- and full-program granularities.

Syntactic vs. Semantic Correctness

Empirically, compilation success proved only weakly indicative of semantic correctness. Many kernels achieved $>90\%$ compilability with $0\%$ correctness on diverse fuzzed inputs.

Figure 2: Partial diff highlighting semantic loss in a Kaizen-fuzzed CUDA to OpenMP translation of the Leaky Integrate-and-Fire neuron model. Despite high compilability, omitted logic in the translation led to silent scientific errors only exposed by input fuzzing.

Compile-Time Error Taxonomy

Analysis of compiler error logs revealed nine major syntactic/structural failure categories in kernel-level translation, including:

• Incorrect loop constructs and invalid OpenMP region nesting (unique to CUDA-to-OpenMP transitions)
• Function signature mismatches, undeclared identifiers, and retained CUDA APIs
• Syntax errors, control-flow violations, type mismatches

For full-program translation, the spectrum broadened to 27 failure categories, including incomplete offloading, broken memory management, residual host/device directives, and missing dependency resolution.

Semantic Bug Taxonomy

Differential testing surfaced six major semantic error categories, all frequently missed by standard testing:

• Intermediate Variable Elimination: Silent dropping of computation steps, typically lexical elimination of intermediates.
• Execution Model Assumption Transfer: Platform-specific parallelism or isolation guarantees incorrectly assumed to carry over.
• Loop Bound Errors: Hardwired loop trip counts mismatched to data and hardware dimensions.
• Missing Statement Faults: Critical in-place mutations dropped, altering accumulative state or outputs.
• Shared Memory Scope Mistranslation: Private/shared variable visibility mismatches, stemming from misaligned platform memory models.
• Multi-dimensional Index Flattening: Loss of multi-dimensional parallel indexing, resulting in silently incorrect coordinate computations.

Crucially, all these errors were input-dependent and evaded detection under developer-provided input suites.

Results and Analysis

Empirical findings can be summarized as follows:

• Neither compilation nor static test success is a reliable indicator of semantic preservation. Several applications pass all static metrics despite producing fundamentally incorrect results on fuzzed inputs.
• Kernel-level translation achieves up to 72% correctness for the best LLMs, but full-program translation remains an open problem. Compilation fails entirely for the best kernel-level model when evaluated end-to-end.
• Fine-tuning LLMs for kernel-only performance can degrade full-program translation capabilities, indicating task overspecialization.
• Semantic errors are often subtle, input-dependent, and model-specific, evading all shallow correctness checks.

Implications and Future Directions

Kaizen's results directly challenge the prevailing practice of using compilation and static testing as sole translation proxies in scientific code porting. The compositional and input-dependent nature of semantic errors in scientific applications, as revealed by systematic fuzzing and differential execution, underscores the need for correctness-oriented evaluation in future LLM translation initiatives. The taxonomies offered by Kaizen provide an actionable blueprint for targeted pre-training, fine-tuning, and evaluation of next-generation LLM-based translation systems.

There are substantial AI and HPC research implications:

• Future LLMs for HPC code translation will require co-training or explicit prompting to reason about complex platform-dependent parallel logic, memory semantics, and nontrivial numerical behaviors that exceed token-level textual equivalence.
• Correctness-focused evaluation frameworks like Kaizen are essential for establishing trust in LLM-generated scientific code, especially as model transparency and code provenance become increasingly consequential.
• The pipeline's structure—combining grammar-based code and input fuzzing, mutation-based data leakage mitigation, and semantic differential analysis—could be generalized for evaluating LLM translations across additional programming model pairs (e.g., CUDA-SYCL, Fortran-C++) or for validating code synthesis in safety-critical domains.

Conclusion

Kaizen establishes a new methodological baseline for evaluating LLM-translated scientific HPC codes, decisively demonstrating that correctness requires adversarial, input-diverse empirical assessment beyond syntactic proxies. The strong application-level and model-specific discrepancies revealed by Kaizen's systematic approach delineate the research frontier for trustworthy LLM-driven code translation in high-consequence computational domains.

---

Reference: "Kaizen: Metamorphic Fuzzing and Differential Testing for LLM-Translated HPC Applications" (2607.04058)