ACCeLLiuM SFT Dataset

• ACCeLLiuM SFT Dataset is a domain-specific supervised fine-tuning corpus mapping data-parallel C/C++ loops to expert-crafted OpenACC directives.
• The dataset comprises 4,033 unique pragma-loop pairs mined from public GitHub repositories with a stratified train/test split and varied directive complexities.
• It serves as a reproducible benchmark for fine-tuning language models, enabling one-line accurate generation of OpenACC pragmas for GPU parallelization.

The ACCeLLiuM SFT dataset is a supervised fine-tuning corpus for automated OpenACC pragma generation from C/C++ data-parallel for loops. It accompanies ACCeLLiuM, a pair of open weights LLMs fine-tuned for generating expert OpenACC directives for GPU parallelization. The dataset contains 4,033 unique pragma-loop pairs mined from public GitHub repositories, with 3,223 pairs for training and 810 for testing. Its central design goal is narrowly defined: given a loop that is already assumed to be a suitable parallelization candidate, the model must generate the appropriate one-line #pragma acc directive that immediately precedes that loop (Jhaveri et al., 20 Sep 2025).

1. Definition and task boundary

ACCeLLiuM SFT is a domain-specific supervision dataset for a constrained code-generation problem: mapping a data-parallel loop to an expert-written OpenACC pragma. In the formulation used by ACCeLLiuM, each example consists of the code to be parallelized and the corresponding pragma to be generated. The dataset therefore targets pragma synthesis, not the broader compiler-analysis problem of determining whether an arbitrary loop is parallelizable (Jhaveri et al., 20 Sep 2025).

This boundary is methodologically important. The dataset does not address loop identification, dependence analysis, or whole-program transformation. Instead, it assumes that the input loop is already an appropriate candidate for OpenACC annotation. The intended model behavior is to reason about data dependencies, memory access patterns, and the selection of the correct directive and clauses—such as reduction, present, copyin, and collapse—and then emit a single precise pragma.

The underlying motivation is the gap between the abstraction offered by directive-based GPU programming and the expertise still required to use it effectively. The paper frames OpenACC as a standard that simplifies GPU programming “to some extent” by abstracting low-level complexity, while still leaving a substantial expert burden in directive selection and clause composition (Jhaveri et al., 20 Sep 2025).

2. Corpus composition and representation

The dataset is organized as a collection of unique pragma-loop pairs in JSONL format, conformant with LLM chat fine-tuning templates. Each item contains a system message specifying expert OpenACC behavior, a user message containing the target insertion point and loop code, and an assistant message containing the reference pragma. In the example provided in the paper, the user prompt begins with <TARGET_PRAGMA_LOCATION> followed by a nested loop, and the assistant output is a single-line directive: #pragma acc parallel loop present(mat[0: size*size]) reduction(+:sum) (Jhaveri et al., 20 Sep 2025).

Property	Value	Notes
Total examples	4,033	Unique pragma-loop pairs
Training set	3,223	80%
Test set	810	20%
Languages	C and C++	Mined from public GitHub repositories
Target unit	for loop + preceding pragma	Only pragma instances immediately before a loop
Storage format	JSONL	Chat-style SFT format

Deduplication is a defining feature of the released corpus. The dataset was reduced to 4,033 unique examples after exact matching on loop bodies, and each pair was assigned a complexity score based on directives and clauses. The stated purpose of this scoring is to promote coverage across a range of pragma complexities rather than to restrict the corpus to a single canonical annotation style.

The representation also encodes a specific interaction pattern for SFT. Because the task is cast as chat fine-tuning rather than plain sequence transduction, the dataset is directly usable in instruction-following code models. A plausible implication is that the dataset was designed to minimize downstream prompt-engineering variance by standardizing the model’s role, input framing, and output length.

3. Mining, extraction, and cleaning pipeline

The collection process begins with the GitHub Code Search API, which was used to find C/C++ files with OpenACC loop directives. This initial mining stage produced 1,509 files containing 30,749 pragma instances (Jhaveri et al., 20 Sep 2025).

The pipeline then narrows its focus to loop pragmas, specifically #pragma acc loop and #pragma acc parallel loop. Source files are parsed into Abstract Syntax Trees (ASTs) using tree-sitter, and each pragma is paired with the immediately succeeding for loop. This extraction stage yielded 25,656 initial pairs.

Subsequent cleaning removes several categories of noise and unsuitable training targets. The paper lists the following filtered-out cases: empty loops, infinite loops, synthetic test cases, and loops with problematic control flow such as break, goto, continue, or return that violate parallelism suitability. After this filtering stage, 10,503 pairs remained. A final deduplication step based on exact matching on loop body reduced the corpus to the released 4,033 unique examples (Jhaveri et al., 20 Sep 2025).

The final organization stage assigns complexity scores, shuffles the examples, and produces a stratified train/test split. This suggests that the dataset was intended not merely as a mined collection but as a reproducible benchmark with controlled variation in pragma difficulty.

4. Complexity profile and directive distribution

The dataset explicitly characterizes pragma complexity by the number of clauses associated with the directive. The distribution reported in the paper is as follows (Jhaveri et al., 20 Sep 2025):

Complexity class	Clause range	Count
Simple	0–2 clauses	1,727
Medium	3–5 clauses	2,290
Complex	6–10 clauses	13
Very Complex	11+ clauses	3

The directive-type distribution is likewise reported in detail. The dominant label is loop with 2,565 instances, followed by parallel with 1,262, kernels with 137, and serial with 22. Rare types include unknown, enter, data, exit, and wait, each with 20 or fewer examples (Jhaveri et al., 20 Sep 2025).

This distribution indicates that the corpus is heavily concentrated on the most common loop-oriented OpenACC forms rather than on the full breadth of directive usage in production accelerator code. The paper does not present this as a defect; rather, it reflects the dataset’s stated purpose of supporting pragma generation for data-parallel loops. A plausible implication is that the benchmark is strongest on loop-local annotation tasks and less suited to evaluating broader OpenACC data-region orchestration.

The complexity profile is also asymmetric. Most examples are simple or medium, while genuinely high-clause pragmas are rare. That imbalance matters when interpreting model performance: exact-match success on the test set is achieved against a label space dominated by relatively common clause combinations, even though the paper also evaluates clause-wise similarity to account for non-canonical but functionally meaningful outputs.

5. Use in supervised fine-tuning

The dataset was used to fine-tune large code models, specifically Llama 3.1 70B and CodeLlama 34B, so that, when given a loop, they generate the appropriate pragma as a one-line response (Jhaveri et al., 20 Sep 2025). In this setup, ACCeLLiuM is not presented as a general-purpose code assistant but as a specialized model family for OpenACC directive synthesis.

The training target is an expert-written pragma immediately preceding the loop. Because the dataset is chat-formatted, the fine-tuning objective aligns with instruction-following model behavior rather than with conventional compiler pass synthesis. The supervision signal therefore encodes both syntactic structure and a specific communicative contract: concise output, no explanatory prose, and direct insertion-readiness.

The paper emphasizes that this task is previously under-served compared to OpenMP. That contextualization is significant in HPC tooling research: OpenACC pragma generation involves not only directive selection but also clause composition that reflects memory movement, reductions, and nested-loop structure. ACCeLLiuM SFT was constructed to expose models to these choices in real code mined from public repositories rather than synthetic teaching examples.

6. Evaluation protocol and empirical outcomes

Evaluation is performed on the held-out test set using multiple complementary metrics: Exact Match Accuracy, Levenshtein Similarity, Directive-Type Match Accuracy, Clause-wise Jaccard Similarity, and Syntactic Validity. Syntactic validity is tested by embedding the generated pragma and loop in a minimal compilable unit (MCU) and compiling it with an OpenACC-compatible compiler (Jhaveri et al., 20 Sep 2025).

The reported results show a large gap between base models and fine-tuned models. For base LLMs (zero-shot), exact-match accuracy is 0% for Llama 3.1 and 0.01 for CodeLlama, directive-type match is 0% for Llama 3.1 and 4–5% for CodeLlama, and the only additional figures reported in the summary are 0.16 Levenshtein Similarity and 0.14 Jaccard Similarity for CodeLlama. By contrast, the ACCeLLiuM-fine-tuned models achieve 43% exact-match accuracy for Llama 3.1 and 50% for CodeLlama, 0.77 and 0.79 Levenshtein similarity respectively, 89% and 87% directive-type match accuracy, and 0.63 and 0.69 clause-wise Jaccard similarity (Jhaveri et al., 20 Sep 2025).

The paper also reports precision/recall/F1 for directive type: 55.2% / 48.8% / 49.4% for Llama 3.1-Finetuned, and 45.2% / 33.8% / 37.5% for CodeLlama-Finetuned. For syntactic validity, the human-written reference pragmas compile successfully in 88.5% of cases, compared with 83.3% for ACCeLLiuM-finetuned Llama 3.1 and 80.9% for ACCeLLiuM-finetuned CodeLlama (Jhaveri et al., 20 Sep 2025).

The abstract summarizes these outcomes in two headline figures: fine-tuned models generate valid pragmas with the correct directive type for $87\%$ of the data-parallel loops and generate exact pragmas—including directives, clauses, clause order, and clause variables—for $50\%$ of the cases (Jhaveri et al., 20 Sep 2025). At the same time, the paper explicitly warns against equating exact string identity with total utility. Even when outputs are not exact matches, they often contain the correct clauses in a different order or include additional clauses that provide finer control over parallel execution, data movement, and concurrency.

A common misconception is therefore that non-exact outputs are necessarily wrong. The evaluation design itself rejects that assumption by including directive-type, Jaccard, and compilation-based metrics alongside exact match. In this framework, exact matching measures canonical agreement with the reference label, while the other metrics capture partial functional adequacy and syntactic usability.

7. Availability, reproducibility, and research significance

The dataset is publicly available as part of the ACCeLLiuM open-source resource bundle, alongside the model code and fine-tuned weights (Jhaveri et al., 20 Sep 2025). The repository identified in the summary is GitHub/OpenACC/ACCeLLiuM. The paper states that the dataset and models are released under open-source licenses, while also noting that the specifics are not stated in the paper.

The release is positioned as a reproducible benchmark for LLM-powered OpenACC pragma generation. That benchmark role is central to its significance. The dataset provides a fixed train/test split, a well-specified mining and cleaning pipeline, complexity stratification, and a multi-metric evaluation protocol. These choices make it suitable not only for direct model training but also for comparative study of code-oriented SFT, instruction formatting, and pragma-generation evaluation.

Its broader stated purpose is to lower the barrier to automated GPU offloading of serially written programs (Jhaveri et al., 20 Sep 2025). Within that framing, ACCeLLiuM SFT is best understood as a specialized HPC dataset that formalizes a previously under-served supervision task: generating OpenACC directives for already-identified data-parallel loops from real public code, at a scale large enough to expose meaningful variation in directive type, clause composition, and syntactic form, but narrow enough to remain a controlled benchmark rather than a general parallelizing compiler corpus.