---
title: 'GiAnt: LLM-Driven Smart-Contract Audit Framework'
url: https://www.emergentmind.com/topics/giant
type: topic
---

# GiAnt: LLM-Driven Smart-Contract Audit Framework

Searching arXiv for the GiAnt framework and closely related smart-contract auditing dataset papers.
GiANT is an LLM-driven framework for automated smart-contract auditing dataset construction, and the GiAnt Corpus is the resulting structured dataset distilled from real-world Code4rena audit reports. The framework is designed to convert heterogeneous professional audit documents into machine-readable records containing vulnerability descriptions, severity labels, impact analyses, proof-of-concept material, mitigation guidance, and code context. In the reported evaluation, GiANT processed 388 real-world audit reports spanning 340 distinct projects and produced a corpus of 7,711 findings across five severity categories, with manual assessment yielding a mean quality score of $4.76 \pm 0.37$ out of 5 and inter-rater agreement $\kappa = 0.88$ [2606.07363].

## 1. Definition and scope

GiANT is defined as **GPT-assisted Auditing Dataset Construction**. Its purpose is not merely vulnerability labeling, but the extraction of structured, audit-grade knowledge from professional smart-contract security reports. The corresponding GiAnt Corpus is therefore broader than a conventional bug benchmark: it includes fields such as vulnerable code, description, impact, proof of concept, mitigation, and gas optimization cases, all derived from real audit findings rather than synthetic templates or tool-generated alerts [2606.07363].

The framework is centered on Code4rena reports. These reports are treated as especially valuable because they contain detailed vulnerability descriptions, severity labels, impact discussions, mitigation suggestions, proof-of-concept details, and code context tied to real-world findings. This design choice places GiANT in the category of report-distillation systems rather than static-analysis aggregators or manually curated vulnerability-only datasets [2606.07363].

A basic conceptual distinction is important. **GiANT** denotes the extraction and curation framework; **GiAnt Corpus** denotes the dataset produced by that framework. The paper treats both as core contributions, with the framework supplying the methodology and the corpus supplying the reusable research artifact [2606.07363].

## 2. Motivation and relation to earlier smart-contract datasets

The project is motivated by two limitations attributed to prior smart-contract auditing datasets. First, manual curation is presented as a scalability bottleneck. The paper cites DAppScan as requiring **22 participants and 44 person-months** to process only **1,199 audit reports**, which is used to argue that purely human-driven curation cannot keep pace with the growth of the smart-contract ecosystem [2606.07363].

Second, earlier datasets are described as too coarse and too narrow. According to the paper, many existing benchmarks emphasize a small set of tool-detectable vulnerability classes such as reentrancy or integer overflow, often using binary or otherwise coarse labels. The paper argues that this omits auditing dimensions that matter in professional practice, including severity level, impact analysis, proof-of-concept evidence, mitigation guidance, and gas optimization [2606.07363].

This framing is tied to a broader change in the threat model. The paper explicitly highlights business logic flaws, access control mistakes, price oracle manipulation, and flash-loan-driven exploits as classes that conventional static tools struggle to detect. Within that framing, LLMs are positioned as potentially useful because they can reason over code semantics and audit narratives, but only if they are supported by a dataset whose structure preserves the richness of expert reports rather than flattening them into binary vulnerability indicators [2606.07363].

A common misconception is that an auditing dataset is interchangeable with a vulnerability detection benchmark. GiANT rejects that equivalence. Its target artifact is multi-dimensional by construction, so it is intended to support not only detection, but also summarization, mitigation recommendation, and gas optimization [2606.07363].

## 3. Framework architecture

GiANT uses a **three-phase pipeline**: **Structured Data Generation**, **Iterative Semantic Refinement**, and **Code Mapping and Scoring Oracle** [2606.07363].

In the first phase, the system crawls publicly available Code4rena reports from **2021 to 2025**. Because the goal is to pair findings with source code, reports without publicly accessible repositories are filtered out. Since the reports appear in heterogeneous formats, PDFs are converted into Markdown using **PyMuPDF**. A regex-based hierarchical parser then exploits the report structure by segmenting documents into severity-based chunks such as High Risk Findings, Medium Risk Findings, Low Risk Findings, Non-Critical Findings, and Gas Optimizations. For each chunk, it isolates individual findings, extracts metadata such as finding id and issue URL, and retains the remainder as a local content block [2606.07363].

The extraction step itself is described as **LLM-based Context-Augmented Semantic Extraction**. It uses local finding text together with external discussion from GitHub issues referenced by the issue URL. The prompting strategy explicitly uses **In-Context Learning (ICL)** and **Chain-of-Thought (CoT)** to populate a JSON schema whose target fields include vulnerable code, description, impact, proof of concept, and mitigation. An **Integrity Validator** immediately checks whether the resulting JSON record is missing fields or is too brief or invalid; deficient records are re-extracted [2606.07363].

The second phase, **Iterative Semantic Refinement**, is designed to improve fidelity and suppress hallucination. A **Logical Consistency Validator** compares the source report and extracted JSON to identify omissions, records discrepancies in a **Quality Assurance (QA) report**, and triggers **Knowledge Recovery** to re-extract missing fragments. Hallucination control is implemented through a text-inclusion algorithm that measures lexical overlap between extracted output $y$ and source text $x$. The paper states that records with
$$
\text{overlap}(y, x) < 0.6
$$
are discarded as hallucination risk [2606.07363].

The principal QA mechanism is an **LLM-as-a-Judge** component. It evaluates extracted records on a **5-point Likert scale** using three criteria: **Accuracy**, **Completeness**, and **No Summarization**. The judge returns a **PASS/FAIL** verdict and a diagnostic critique. Failed records enter a recursive repair loop in which a knowledge rescuer synthesizes local report context and external issue discussion, the extractor reruns with rejection history, and the judge reevaluates the revised record. After up to **3 retries**, unrecoverable records are discarded. The paper summarizes this repair cycle with notation of the form
$$
d_{\text{new}} \leftarrow \text{LLM\_Extract}(C, r, H),
$$
$$
(v, \rho) \leftarrow \text{LLM\_Judge}(d_{\text{new}}, C),
$$
where $v \in \{\text{PASS, FAIL}\}$ and $\rho$ is the critique [2606.07363].

The third phase addresses source-code grounding. Because extracted vulnerable-code snippets are often incomplete or linked unstably, GiANT applies a **dual-path mapping strategy**. It performs **precise mapping** when reports contain explicit GitHub anchors such as `#L100-L110`, and **inference-based mapping** otherwise, with the LLM resolving target filenames from vulnerability context, searching within relevant files, and falling back to global search if necessary. Directories such as `test`, `mock`, and `node_modules` are filtered out. A subsequent LLM-based **Alignment Discriminator** scores semantic consistency between retrieved code and the documented description and impact on a 5-point Likert scale; only records with score **$\ge 4$** are retained [2606.07363].

## 4. Corpus scale and composition

The evaluated corpus is drawn from **388** real-world audit reports and represents **340** distinct projects. The final GiAnt Corpus contains **7,711** findings distributed across five severity categories [2606.07363].

| Severity category | Findings |
|---|---:|
| High Risk | 1,048 |
| Medium Risk | 2,443 |
| Low Risk | 782 |
| Non-Critical | 723 |
| Gas Optimization | 2,715 |

This composition is significant because it makes the corpus simultaneously a security dataset and an efficiency dataset. Gas optimization accounts for **2,715** records, so the corpus is not restricted to exploit-oriented failure modes. The paper explicitly presents this as part of the corpus’s broader usefulness for smart contract engineering [2606.07363].

The construction cost is also reported. GiANT consumed about **\$597.2 USD**, **21.3 million tokens**, and an average of **62.6k tokens per report**. These values situate the framework between purely manual curation and unconstrained large-scale LLM processing: the system is automated, but its extraction, QA, and recursive repair stages remain computationally nontrivial [2606.07363].

## 5. Quality control and empirical reliability

The paper’s central empirical claim is that the extraction pipeline achieves high fidelity relative to human judgment. Manual evaluation was conducted on **400 sampled records** stratified across the five severity categories. The first two authors manually reviewed them; **350** were independently split for assessment, and **50** were jointly reviewed for inter-coder agreement [2606.07363].

The reported outcome is a **mean quality score of $4.76 \pm 0.37$ out of 5** and **inter-rater agreement $\kappa = 0.88$**. The paper characterizes this as near-perfect agreement. It also invokes a Cochran-style finite population argument to justify the sample size, claiming approximately **95% confidence** with a margin of error around **4.75%** [2606.07363].

The evaluation protocol reflects the framework’s extractive rather than abstractive orientation. The judge criteria include **No Summarization**, meaning that the desired output is not a compressed paraphrase of the source report, but a faithful structured preservation of expert audit content. This is an important methodological point because many LLM pipelines optimize for concise reformulation; GiANT instead prioritizes information retention and traceability to source text [2606.07363].

Another misconception addressed implicitly by the results is that lexical or semantic similarity alone suffices to validate extraction. GiANT combines overlap filtering, logical consistency validation, recursive repair, and LLM judging. This suggests that no single QA signal is treated as adequate in isolation; reliability is produced by layered validation rather than by one metric [2606.07363].

## 6. Benchmark tasks, observed model limits, and research significance

To demonstrate the corpus’s practical utility, the paper benchmarks **four state-of-the-art LLMs**: **GPT-3.5-turbo**, **GPT-4o**, **Qwen3-Coder-480B-A35B-Instruct**, and **DeepSeek-V3**. These models are evaluated on **vulnerability detection**, **code summarization**, **mitigation recommendation**, and **automated gas optimization** [2606.07363].

For vulnerability detection, the tasks are framed as binary and multi-class classification, evaluated with **Accuracy**, **Precision**, **Recall**, and **F1**. The paper reports that the models perform reasonably in binary detection, but that performance collapses on severity classification, with multi-class F1 scores only about **0.280–0.302**. The result is significant because severity assignment is one of the expert judgments that many existing datasets either omit or oversimplify [2606.07363].

For code summarization, models generate natural-language explanations of vulnerable code and perform risk or impact analysis. The evaluation uses **BERTScore** and an **LLM-judge Likert score**. The paper reports relatively high BERTScore values but low expert-like judge scores, indicating that semantic resemblance does not necessarily imply faithful capture of audit logic [2606.07363].

For mitigation recommendation, the same metrics are used. This is reported as the strongest task for the evaluated models: all achieve high semantic similarity and relatively strong judge scores, with **GPT-4o**, **Qwen3-Coder**, and **DeepSeek-V3** around the **4.0** range. Even here, however, output variance remains high [2606.07363].

For gas optimization, the models refactor code while attempting to preserve functionality and improve gas efficiency. The paper concludes that LLMs show a foundational ability on this task, but that quality is unstable and highly variable. Across tasks, the paper’s synthesized conclusion is that **zero-shot prompting is not sufficient for professional smart contract auditing** [2606.07363].

Taken together, these benchmarks position the GiAnt Corpus as both a dataset and a diagnostic instrument. It is large enough and structured enough to expose a gap between plausible language generation and professional auditing competence. A plausible implication is that future progress will depend less on generic zero-shot prompting and more on fine-tuning, retrieval augmentation, and task-specific reasoning systems, which the paper identifies as natural next steps [2606.07363].

Source: https://www.emergentmind.com/topics/giant