Validation-Centric AI-Assisted GPU Porting of a 250,000+ Line Legacy Weather Simulation Code

Abstract: Recent advances in LLMs have made CLI-based AI agents a practical tool for accelerating GPU porting of large legacy scientific applications. Such applications, however, are not merely old code bases; they are scientific assets whose credibility has been accumulated through long-term development, comparison with observations, and use in domain studies. GPU porting must therefore preserve this scientific validity while adapting the implementation to GPU-centric HPC systems. This paper presents a validation-centric AI-assisted GPU porting workflow through a case study of CReSS, a legacy Fortran weather simulation code with more than 250,000 lines. The workflow uses an AI agent to extract OpenMP regions, generate dump-based kernel benchmarks from physically meaningful simulation states, apply OpenACC transformations, and validate results through element-wise comparison with dumped reference data and application-level validation. Using a real typhoon simulation, the workflow produced numerically validated GPU implementations for 162 target kernels and achieved a 5.1x application-level speedup within practical wall-clock development cost. In particular, it detected numerical discrepancies in five kernels caused by floating-point and intrinsic-function differences, including threshold-sensitive branch divergence and cancellation effects, enabling feedback to the application developers. The case study suggests that, for large legacy scientific applications requiring dump-based validation, practical AI-assisted GPU porting must manage session-spanning context, runtime-state reconstruction, and costly recovery from small static-analysis omissions. These findings demonstrate that AI-assisted GPU porting requires not only code generation, but validation-centric workflow design.

• The paper presents a validation-centric AI workflow that extracted and ported 162 active kernels from the 260,000-line CReSS weather model, combining runtime-state reconstruction, CPU reference benchmarks, kernel-level comparisons, and integrated application testing.
• The paper found five interpretable numerical discrepancies caused by rounding, intrinsic-function differences, cancellation, and threshold-sensitive branches, while the GPU simulation met application-level pressure-error criteria below 10⁻⁴.
• The paper achieved a 5.1-fold application-level speedup, reducing median timestep time from 9.51 to 1.88 seconds on an H100-equipped node, but shows that domain expertise and explicit recovery procedures remain essential for trustworthy AI-assisted porting.

Problem formulation and contribution

The paper addresses GPU porting of a class of applications for which conventional software-engineering criteria are insufficient. CReSS, the Cloud Resolving Storm Simulator, is a Fortran weather model with 599 Fortran 90 source files, approximately 260,000 lines of code, and 387 OpenMP parallel regions. Its scientific credibility derives from sustained development, observational comparison, and repeated use in meteorological studies. Consequently, the objective is not merely to generate syntactically valid accelerator code or maximize isolated-kernel throughput. It is to obtain a GPU implementation whose numerical behavior remains scientifically acceptable and whose deviations from the CPU implementation can be localized and interpreted.

The paper presents a validation-centric workflow in which a CLI-based AI agent assists code inspection, kernel extraction, benchmark generation, OpenACC transformation, integration, and debugging. Validation is retained as the controlling constraint. The workflow produced GPU implementations for 162 kernels exercised by a realistic typhoon scenario, achieved a 5.1-fold application-level speedup, and identified five numerical discrepancies requiring domain-level interpretation (2608.13122).

The central claim is therefore narrower and more defensible than a claim of autonomous GPU porting: AI assistance can reduce the manual burden of a validation-preserving porting process, but it does not eliminate the need for runtime-state reconstruction, numerical diagnosis, human judgment, or explicit workflow control.

Validation requirements for scientific GPU porting

The paper distinguishes verification of generated transformations from validation of numerical behavior. Verification concerns whether an extracted or transformed kernel reproduces the expected behavior of the original implementation. Validation concerns whether CPU–GPU numerical differences are acceptable for the scientific application. This distinction is important because successful compilation, plausible OpenACC directives, and even agreement on selected application outputs do not establish correctness.

A full application treated as a black box is inadequate for diagnosing failures. If an integrated simulation diverges, the responsible kernel, transformation, or numerical mechanism may be difficult to identify. The workflow consequently decomposes the application into diagnostic units and compares each unit against CPU reference data before integration. This decomposition also bounds the amount of source code, execution state, and validation output handled in an individual agent session.

The need for kernel-level validation is amplified by GPU execution semantics. GPU transformations can change reduction order, expose inner-loop parallelism, alter rounding behavior, and invoke different implementations of transcendental functions. Bitwise disagreement is therefore not automatically an implementation defect. Conversely, it cannot be dismissed categorically: in a time-dependent weather model, a difference of one unit in the last place can change a threshold comparison and alter subsequent physical updates.

The workflow uses both kernel-level and application-level tests. Kernel-level tests localize discrepancies; application-level tests determine whether the integrated simulation remains within domain-defined acceptance criteria. The paper’s results demonstrate why both are necessary: all 162 kernels were processed through local validation, yet integration still exposed a branch omitted from a benchmark because it was inactive at the selected snapshot.

CReSS case study and workflow architecture

The validation scenario is a western Pacific typhoon simulation initialized from real Grid Point Value data. It uses a $899 \times 899 \times 128$ grid, approximately 100 million grid points, a horizontal resolution of about 2 km, and enabled cloud microphysics, radiation, turbulence, and surface processes. The simulation covers 30 minutes, corresponding to 360 timesteps. This duration is shorter than a production forecast but long enough to exercise coupled physical processes and allow numerical differences to accumulate.

Of the 387 OpenMP regions, 162 execute in this scenario and become GPU-porting targets. The workflow comprises six phases:

1. Code meta-review identifies OpenMP regions, calls, synchronization constructs, thread-ID usage, global-state writes, and candidate barriers to directive-based transformation.
2. Profiling determines invocation counts and the active target-kernel set.
3. Kernel extraction and CPU benchmark generation produces dump-instrumented CPU code, runtime-state data, and standalone CPU benchmarks.
4. Kernel-level GPU transformation converts verified CPU benchmarks to OpenACC implementations and compares their outputs with CPU references.
5. Integration inserts validated GPU kernels into the original application using conditional compilation.
6. Performance validation uses Nsight to inspect execution time, memory-bandwidth utilization, and Unified Memory behavior, followed by kernel-local revisions where necessary.

The workflow’s artifact structure is as important as its agent prompts. Variable lists, compiler options, binary-I/O conventions, dump rules, validation tolerances, and recovery procedures are externalized into persistent specifications. This design treats the workflow itself as an engineering object rather than assuming that conversational context will remain stable across interactive sessions.

Runtime-state reconstruction and dump-based benchmarks

The most technically consequential design choice is the use of runtime states generated by the original CPU simulation. Synthetic or random kernel inputs are inadequate because CReSS kernels operate on states produced by initialization, time-dependent physical processes, conditional allocation, and configuration-specific execution paths. A benchmark constructed from artificial data may compile and execute while failing to exercise the numerical regimes and branch conditions that occur in a real typhoon simulation.

For each target region, the CPU simulation is instrumented to dump kernel inputs and reference outputs. The selected dump point is the final invocation of the kernel during the 360-step scenario. The agent extracts the required variables and generates an independent benchmark that replays the dumped state. The CPU benchmark is first compared with the dumped CPU reference; only then is it transformed to OpenACC and compared against the same reference.

The element-wise validation metric uses a relative error normalized by the larger of the reference magnitude and $10^{-20}$, with a single-precision tolerance of $10^{-5}$. This arrangement separates failures in extraction and replay from failures introduced by GPU transformation. A CPU benchmark that fails indicates a problem in variable enumeration, dump instrumentation, data reconstruction, or replay. A GPU benchmark that fails initiates local diagnosis of OpenACC directives, loop-independence assumptions, and numerical differences.

This design also exposes a fundamental cost asymmetry. A local static-analysis omission can invalidate a dump run that requires approximately one hour of simulation re-execution. In the full workflow, more than 2,000 dump insertion points were required. The reduced experimental setting contained 15 kernels, 690 variable-list entries, 134 condition-dependent entries, and approximately 89 GB of dump data. Thus, the expensive operation is often not GPU code generation but the acquisition and repair of scientifically meaningful runtime state.

GPU transformation and numerical validation

The agent derives an OpenACC benchmark from the verified CPU benchmark, generally using kernels regions and loop independent annotations. The latter are treated as candidate assertions rather than proofs of independence. Existing OpenMP parallelization provides evidence for parallelism, but it does not guarantee that every loop is free of vertical dependencies, sequential accumulations, boundary hazards, or state-dependent constraints.

The immediate dump-based comparison limits the risk of incorrect independence assumptions. If a loop marked independent changes the output, the transformation can be revised, for example by marking the loop sequential. This is a practical compromise: aggressive transformation is permitted, but only within a workflow that detects its consequences before integration.

The initial implementation uses NVIDIA Unified Memory through -gpu=managed rather than explicit OpenACC data regions. This simplifies the first port and avoids requiring the agent to manage inter-kernel data lifetimes manually. The choice also constrains performance interpretation. Explicit data movement, asynchronous execution, kernel fusion, communication overlap, and data-structure changes are excluded from the study because they require additional validation mechanisms.

The five unresolved kernel-level discrepancies are particularly informative:

Kernel	Numerical mechanism	Observed effect
bruntv.f90	Threshold-sensitive branch after exp() and log()	A one-ulp difference changed t <= tlow
disptke.f90	Intrinsic-function differences and cancellation	Intermediate values differed by approximately 25 ulps
cloudcov.f90	Relative error near an exactly zero reference	Absolute difference was $1.9 \times 10^{-8}$
siadjst.f90	Threshold-sensitive microphysics condition	qi > dqi changed branch selection
swadjst.f90	Threshold-sensitive microphysics condition	qc > dqc changed branch selection

In bruntv.f90, the CPU produced $233.16002$ K and the GPU produced $233.16000$ K in single precision. The resulting difference changed whether a latent-heat correction was applied. In disptke.f90, the larger discrepancy was associated with intrinsic-function implementation differences and cancellation rather than ordinary one-ulp rounding. These cases were judged acceptable by the CReSS developers, but the paper appropriately does not treat that judgment as automatable.

The implication is significant: a tolerance-based pass/fail test is insufficient for scientific validation. Numerical discrepancies must be classified by mechanism, location, branch sensitivity, and application consequence. The workflow provides the evidence for that classification, while domain experts retain responsibility for deciding acceptability.

Integrated validation and performance

After kernel-level validation, the GPU kernels are integrated into the original application using conditional compilation. Per-kernel or per-kernel-group switches enable bisection-style diagnosis when application-level validation fails. This is particularly useful because snapshot validation does not cover every state encountered during full time evolution.

The integrated GPU application completes the 360-step typhoon simulation and satisfies the pressure-perturbation criteria. The reported normalized errors are:

• $a_1 = 1.0 \times 10^{-5}$ for the maximum pressure perturbation.
• $a_2 = 5.6 \times 10^{-5}$ for the minimum pressure perturbation.

Both satisfy the stated threshold of $10^{-4}$. This establishes application-level agreement for the selected scenario, but not universal equivalence across all CReSS configurations or meteorological cases.

Performance is measured on one Miyabi-G node containing a 72-core Grace CPU and an NVIDIA H100 GPU. The CPU baseline uses 72 OpenMP threads; the GPU version uses OpenACC and Unified Memory. Median timestep execution decreases from 9.51 seconds on the CPU to 1.88 seconds on the GPU, yielding a 5.1-fold speedup.

The result is below the approximately 8-fold ratio between nominal H100 HBM and Grace CPU memory bandwidth, but the comparison is not a hardware-bandwidth ceiling test. The implementation uses directive-generated GPU code, Unified Memory, and deliberately avoids aggressive inter-kernel optimization. Across all 162 kernels, measured memory-bandwidth utilization is approximately 35–60% of the GPU peak, with no dominant unintended CPU execution or pathological Unified Memory migration observed.

Figure 1: Top 10 GPU kernels by measured execution time, with colors indicating memory-bandwidth efficiency relative to the GPU peak.

The performance result should therefore be interpreted as evidence that the validation-centric workflow can produce a useful GPU implementation, not as evidence that the resulting code is fully optimized. The study’s priority is validated acceleration under practical development constraints.

Workflow cost and agent-dependent failure modes

The paper reports approximately 100 hours of interactive GPU-node time over about three months, including exploration, failed attempts, and recovery. This is not presented as a controlled comparison against manual porting, and the sessions were human-supervised. Nevertheless, it quantifies the operational scale of the case study.

More than half of the development wall-clock time was spent in kernel extraction and CPU benchmark generation. Once a standalone benchmark was correctly constructed, OpenACC transformation and local testing were comparatively inexpensive. The dominant bottleneck was therefore the construction of a valid interface between the original simulation’s runtime state and the independent benchmark.

A representative failure involves condition-dependent dummy allocations. When a physics option is disabled, a full-domain array may be represented by a minimal dummy object. A callee can still expose a full-domain argument interface, while the caller prevents invalid accesses through configuration-dependent control flow. Local inspection of the target region may consequently suggest that dumping the array is ordinary and valid, although dumping it as a full-domain object causes a segmentation fault.

The paper mitigates omissions in variable enumeration by applying default(none) to copied OpenMP regions. Compiler diagnostics expose undeclared data-sharing attributes, converting some expensive runtime failures into cheaper compile-time feedback. This is a strong workflow design choice because it uses the compiler as a conservative dependency-discovery mechanism rather than relying exclusively on agentic static analysis.

The controlled comparison between prompt-only continuation and persistent specifications shows a clear convergence effect. In five reduced Phase-3 runs, prompt-based continuation converged in three cases, whereas specification-based continuation converged in all five. Persistent specifications preserved compiler options, byte-order settings, benchmark configuration rules, interactive-job requirements, and recovery policies across session boundaries.

However, persistent specifications did not minimize execution cost. Specification-based runs required between one and seven dump executions. Adding an explicit batch-checking recovery rule reduced successful runs to one through three dump executions, excluding one job-control failure. The result supports a precise conclusion: externalized specifications improve procedural consistency, while cost-aware recovery is separately required to avoid repeated high-cost simulation execution.

Limitations and open questions

The validation coverage is scenario-specific. The 162 kernels are those exercised by one 30-minute typhoon simulation; the remaining OpenMP regions are not validated. The GPU implementation is therefore not a complete certification of all CReSS execution paths.

Snapshot-based kernel validation is also incomplete. Selecting the final invocation of each kernel makes dump reuse practical, but it can omit branches active at intermediate timesteps. The paper reports an integrated failure caused by a CPU benchmark that omitted such a branch. The benchmark had passed its selected snapshot, but full-application execution later exercised the missing path. Multiple dump points or multi-scenario validation would improve coverage, but the associated storage cost is substantial: even a minimal configuration generated more than 400 GB of dump data.

The application-level acceptance test uses only maximum and minimum pressure perturbations. These metrics are scientifically meaningful for the selected case, but they cannot characterize every possible divergence in fields, physical tendencies, conservation properties, or downstream diagnostic quantities. The paper does not claim that agreement in these two metrics establishes comprehensive meteorological equivalence.

The performance evaluation is likewise bounded. It uses one GH200 node, single precision, Unified Memory, and a single-node execution model. Multi-node scaling, explicit data regions, asynchronous execution, kernel fusion, communication overlap, and broader memory-management optimization remain unassessed. The study also evaluates a workflow using Claude Code Opus 4.5–4.6 rather than establishing model-independent behavior. The reported agent failure modes should therefore be understood as observations about the interaction between a particular agentic workflow and HPC execution costs, not as universal properties of LLMs.

Open questions remain about how to select representative dump points systematically, how to compress and manage reference-data lifecycles, how to propagate validation across multiple meteorological scenarios, and how to extend the same diagnostic discipline to optimizations that alter inter-kernel data movement or execution ordering.

Conclusion

The paper presents AI-assisted GPU porting as a validation and workflow-engineering problem rather than a code-generation problem. Applied to a 260,000-line Fortran weather model, the workflow validated 162 GPU kernels, detected five interpretable numerical discrepancies, satisfied application-level criteria for a 360-step typhoon simulation, and achieved a 5.1-fold speedup over a 72-thread Grace CPU baseline (2608.13122).

Its principal result is methodological: AI agents are useful when embedded in a process that supplies physically meaningful runtime states, localized reference comparisons, persistent specifications, conditional integration, and explicit recovery policies. The case study also demonstrates that the main practical costs arise from runtime-state reconstruction and repeated simulation execution. Kernel transformation is only one component of trustworthy GPU porting; maintaining scientific validity requires a structured validation pipeline in which automation accelerates artifact production but does not replace numerical interpretation or domain-specific judgment.