---
title: ACCeLLiuM SFT Dataset
url: https://www.emergentmind.com/topics/accellium-sft-dataset
type: topic
---

# ACCeLLiuM SFT Dataset

The **ACCeLLiuM SFT dataset** is a supervised fine-tuning corpus for automated **OpenACC pragma generation** from **C/C++ data-parallel `for` loops**. It accompanies ACCeLLiuM, a pair of open weights large language models fine-tuned for generating expert OpenACC directives for GPU parallelization. The dataset contains **4,033 unique pragma-loop pairs** mined from public GitHub repositories, with **3,223 pairs for training** and **810 for testing**. Its central design goal is narrowly defined: given a loop that is already assumed to be a suitable parallelization candidate, the model must generate the appropriate one-line `#pragma acc` directive that immediately precedes that loop [2509.20380].

## 1. Definition and task boundary

ACCeLLiuM SFT is a **domain-specific supervision dataset** for a constrained code-generation problem: mapping a data-parallel loop to an expert-written OpenACC pragma. In the formulation used by ACCeLLiuM, each example consists of the code to be parallelized and the corresponding pragma to be generated. The dataset therefore targets **pragma synthesis**, not the broader compiler-analysis problem of determining whether an arbitrary loop is parallelizable [2509.20380].

This boundary is methodologically important. The dataset does **not** address loop identification, dependence analysis, or whole-program transformation. Instead, it assumes that the input loop is already an appropriate candidate for OpenACC annotation. The intended model behavior is to reason about **data dependencies**, **memory access patterns**, and the selection of the correct directive and clauses—such as `reduction`, `present`, `copyin`, and `collapse`—and then emit a single precise pragma.

The underlying motivation is the gap between the abstraction offered by directive-based GPU programming and the expertise still required to use it effectively. The paper frames OpenACC as a standard that simplifies GPU programming “to some extent” by abstracting low-level complexity, while still leaving a substantial expert burden in directive selection and clause composition [2509.20380].

## 2. Corpus composition and representation

The dataset is organized as a collection of **unique pragma-loop pairs** in **JSONL** format, conformant with **LLM chat fine-tuning templates**. Each item contains a `system` message specifying expert OpenACC behavior, a `user` message containing the target insertion point and loop code, and an `assistant` message containing the reference pragma. In the example provided in the paper, the user prompt begins with `<TARGET_PRAGMA_LOCATION>` followed by a nested loop, and the assistant output is a single-line directive:
`#pragma acc parallel loop present(mat[0: size*size]) reduction(+:sum)` [2509.20380].

| Property | Value | Notes |
|---|---:|---|
| Total examples | 4,033 | Unique pragma-loop pairs |
| Training set | 3,223 | 80% |
| Test set | 810 | 20% |
| Languages | C and C++ | Mined from public GitHub repositories |
| Target unit | `for` loop + preceding pragma | Only pragma instances immediately before a loop |
| Storage format | JSONL | Chat-style SFT format |

Deduplication is a defining feature of the released corpus. The dataset was reduced to **4,033 unique examples** after exact matching on loop bodies, and each pair was assigned a **complexity score** based on directives and clauses. The stated purpose of this scoring is to promote coverage across a range of pragma complexities rather than to restrict the corpus to a single canonical annotation style.

The representation also encodes a specific interaction pattern for SFT. Because the task is cast as chat fine-tuning rather than plain sequence transduction, the dataset is directly usable in instruction-following code models. A plausible implication is that the dataset was designed to minimize downstream prompt-engineering variance by standardizing the model’s role, input framing, and output length.

## 3. Mining, extraction, and cleaning pipeline

The collection process begins with the **GitHub Code Search API**, which was used to find **C/C++ files with OpenACC loop directives**. This initial mining stage produced **1,509 files** containing **30,749 pragma instances** [2509.20380].

The pipeline then narrows its focus to **loop pragmas**, specifically `#pragma acc loop` and `#pragma acc parallel loop`. Source files are parsed into **Abstract Syntax Trees (ASTs)** using **tree-sitter**, and each pragma is paired with the **immediately succeeding `for` loop**. This extraction stage yielded **25,656 initial pairs**.

Subsequent cleaning removes several categories of noise and unsuitable training targets. The paper lists the following filtered-out cases: **empty loops**, **infinite loops**, **synthetic test cases**, and loops with **problematic control flow** such as `break`, `goto`, `continue`, or `return` that violate parallelism suitability. After this filtering stage, **10,503** pairs remained. A final deduplication step based on **exact matching on loop body** reduced the corpus to the released **4,033 unique examples** [2509.20380].

The final organization stage assigns **complexity scores**, shuffles the examples, and produces a **stratified train/test split**. This suggests that the dataset was intended not merely as a mined collection but as a reproducible benchmark with controlled variation in pragma difficulty.

## 4. Complexity profile and directive distribution

The dataset explicitly characterizes pragma complexity by the number of clauses associated with the directive. The distribution reported in the paper is as follows [2509.20380]:

| Complexity class | Clause range | Count |
|---|---|---:|
| Simple | 0–2 clauses | 1,727 |
| Medium | 3–5 clauses | 2,290 |
| Complex | 6–10 clauses | 13 |
| Very Complex | 11+ clauses | 3 |

The directive-type distribution is likewise reported in detail. The dominant label is **`loop`** with **2,565** instances, followed by **`parallel`** with **1,262**, **`kernels`** with **137**, and **`serial`** with **22**. Rare types include **`unknown`**, **`enter`**, **`data`**, **`exit`**, and **`wait`**, each with **20 or fewer** examples [2509.20380].

This distribution indicates that the corpus is heavily concentrated on the most common loop-oriented OpenACC forms rather than on the full breadth of directive usage in production accelerator code. The paper does not present this as a defect; rather, it reflects the dataset’s stated purpose of supporting **pragma generation for data-parallel loops**. A plausible implication is that the benchmark is strongest on loop-local annotation tasks and less suited to evaluating broader OpenACC data-region orchestration.

The complexity profile is also asymmetric. Most examples are **simple** or **medium**, while genuinely high-clause pragmas are rare. That imbalance matters when interpreting model performance: exact-match success on the test set is achieved against a label space dominated by relatively common clause combinations, even though the paper also evaluates clause-wise similarity to account for non-canonical but functionally meaningful outputs.

## 5. Use in supervised fine-tuning

The dataset was used to fine-tune large code models, specifically **Llama 3.1 70B** and **CodeLlama 34B**, so that, when given a loop, they generate the appropriate pragma as a **one-line response** [2509.20380]. In this setup, ACCeLLiuM is not presented as a general-purpose code assistant but as a specialized model family for OpenACC directive synthesis.

The training target is an expert-written pragma immediately preceding the loop. Because the dataset is chat-formatted, the fine-tuning objective aligns with instruction-following model behavior rather than with conventional compiler pass synthesis. The supervision signal therefore encodes both syntactic structure and a specific communicative contract: concise output, no explanatory prose, and direct insertion-readiness.

The paper emphasizes that this task is **previously under-served compared to OpenMP**. That contextualization is significant in HPC tooling research: OpenACC pragma generation involves not only directive selection but also clause composition that reflects memory movement, reductions, and nested-loop structure. ACCeLLiuM SFT was constructed to expose models to these choices in real code mined from public repositories rather than synthetic teaching examples.

## 6. Evaluation protocol and empirical outcomes

Evaluation is performed on the **held-out test set** using multiple complementary metrics: **Exact Match Accuracy**, **Levenshtein Similarity**, **Directive-Type Match Accuracy**, **Clause-wise Jaccard Similarity**, and **Syntactic Validity**. Syntactic validity is tested by embedding the generated pragma and loop in a **minimal compilable unit (MCU)** and compiling it with an **OpenACC-compatible compiler** [2509.20380].

The reported results show a large gap between base models and fine-tuned models. For **base LLMs (zero-shot)**, exact-match accuracy is **0% for Llama 3.1** and **0.01 for CodeLlama**, directive-type match is **0% for Llama 3.1** and **4–5% for CodeLlama**, and the only additional figures reported in the summary are **0.16 Levenshtein Similarity** and **0.14 Jaccard Similarity** for CodeLlama. By contrast, the **ACCeLLiuM-fine-tuned** models achieve **43% exact-match accuracy for Llama 3.1** and **50% for CodeLlama**, **0.77** and **0.79** Levenshtein similarity respectively, **89%** and **87%** directive-type match accuracy, and **0.63** and **0.69** clause-wise Jaccard similarity [2509.20380].

The paper also reports **precision/recall/F1** for directive type: **55.2% / 48.8% / 49.4%** for **Llama 3.1-Finetuned**, and **45.2% / 33.8% / 37.5%** for **CodeLlama-Finetuned**. For **syntactic validity**, the **human-written reference pragmas** compile successfully in **88.5%** of cases, compared with **83.3%** for **ACCeLLiuM-finetuned Llama 3.1** and **80.9%** for **ACCeLLiuM-finetuned CodeLlama** [2509.20380].

The abstract summarizes these outcomes in two headline figures: fine-tuned models generate **valid pragmas with the correct directive type for $87\%$** of the data-parallel loops and generate **exact pragmas—including directives, clauses, clause order, and clause variables—for $50\%$** of the cases [2509.20380]. At the same time, the paper explicitly warns against equating exact string identity with total utility. Even when outputs are not exact matches, they often contain the **correct clauses in a different order** or include **additional clauses** that provide finer control over parallel execution, data movement, and concurrency.

A common misconception is therefore that non-exact outputs are necessarily wrong. The evaluation design itself rejects that assumption by including **directive-type**, **Jaccard**, and **compilation-based** metrics alongside exact match. In this framework, exact matching measures canonical agreement with the reference label, while the other metrics capture partial functional adequacy and syntactic usability.

## 7. Availability, reproducibility, and research significance

The dataset is **publicly available** as part of the **ACCeLLiuM open-source resource bundle**, alongside the model code and fine-tuned weights [2509.20380]. The repository identified in the summary is **GitHub/OpenACC/ACCeLLiuM**. The paper states that the dataset and models are released under **open-source licenses**, while also noting that the **specifics are not stated in the paper**.

The release is positioned as a **reproducible benchmark** for **LLM-powered OpenACC pragma generation**. That benchmark role is central to its significance. The dataset provides a fixed train/test split, a well-specified mining and cleaning pipeline, complexity stratification, and a multi-metric evaluation protocol. These choices make it suitable not only for direct model training but also for comparative study of code-oriented SFT, instruction formatting, and pragma-generation evaluation.

Its broader stated purpose is to **lower the barrier to automated GPU offloading of serially written programs** [2509.20380]. Within that framing, ACCeLLiuM SFT is best understood as a specialized HPC dataset that formalizes a previously under-served supervision task: generating OpenACC directives for already-identified data-parallel loops from real public code, at a scale large enough to expose meaningful variation in directive type, clause composition, and syntactic form, but narrow enough to remain a controlled benchmark rather than a general parallelizing compiler corpus.

Source: https://www.emergentmind.com/topics/accellium-sft-dataset