GiAnt: LLM-Driven Smart-Contract Audit Framework

• GiAnt is an LLM-assisted framework that converts heterogeneous smart-contract audit reports into structured, machine-readable records.
• It employs a three-phase pipeline—data extraction, semantic refinement, and code mapping—to ensure high fidelity with a mean quality score of 4.76/5.
• The resulting GiAnt Corpus comprises 7,711 findings across 340 projects, supporting advanced vulnerability detection, mitigation, and gas optimization.

Searching arXiv for the GiAnt framework and closely related smart-contract auditing dataset papers. GiANT is an LLM-driven framework for automated smart-contract auditing dataset construction, and the GiAnt Corpus is the resulting structured dataset distilled from real-world Code4rena audit reports. The framework is designed to convert heterogeneous professional audit documents into machine-readable records containing vulnerability descriptions, severity labels, impact analyses, proof-of-concept material, mitigation guidance, and code context. In the reported evaluation, GiANT processed 388 real-world audit reports spanning 340 distinct projects and produced a corpus of 7,711 findings across five severity categories, with manual assessment yielding a mean quality score of $4.76 \pm 0.37$ out of 5 and inter-rater agreement $\kappa = 0.88$ (Zhang et al., 5 Jun 2026).

1. Definition and scope

GiANT is defined as GPT-assisted Auditing Dataset Construction. Its purpose is not merely vulnerability labeling, but the extraction of structured, audit-grade knowledge from professional smart-contract security reports. The corresponding GiAnt Corpus is therefore broader than a conventional bug benchmark: it includes fields such as vulnerable code, description, impact, proof of concept, mitigation, and gas optimization cases, all derived from real audit findings rather than synthetic templates or tool-generated alerts (Zhang et al., 5 Jun 2026).

The framework is centered on Code4rena reports. These reports are treated as especially valuable because they contain detailed vulnerability descriptions, severity labels, impact discussions, mitigation suggestions, proof-of-concept details, and code context tied to real-world findings. This design choice places GiANT in the category of report-distillation systems rather than static-analysis aggregators or manually curated vulnerability-only datasets (Zhang et al., 5 Jun 2026).

A basic conceptual distinction is important. GiANT denotes the extraction and curation framework; GiAnt Corpus denotes the dataset produced by that framework. The paper treats both as core contributions, with the framework supplying the methodology and the corpus supplying the reusable research artifact (Zhang et al., 5 Jun 2026).

2. Motivation and relation to earlier smart-contract datasets

The project is motivated by two limitations attributed to prior smart-contract auditing datasets. First, manual curation is presented as a scalability bottleneck. The paper cites DAppScan as requiring 22 participants and 44 person-months to process only 1,199 audit reports, which is used to argue that purely human-driven curation cannot keep pace with the growth of the smart-contract ecosystem (Zhang et al., 5 Jun 2026).

Second, earlier datasets are described as too coarse and too narrow. According to the paper, many existing benchmarks emphasize a small set of tool-detectable vulnerability classes such as reentrancy or integer overflow, often using binary or otherwise coarse labels. The paper argues that this omits auditing dimensions that matter in professional practice, including severity level, impact analysis, proof-of-concept evidence, mitigation guidance, and gas optimization (Zhang et al., 5 Jun 2026).

This framing is tied to a broader change in the threat model. The paper explicitly highlights business logic flaws, access control mistakes, price oracle manipulation, and flash-loan-driven exploits as classes that conventional static tools struggle to detect. Within that framing, LLMs are positioned as potentially useful because they can reason over code semantics and audit narratives, but only if they are supported by a dataset whose structure preserves the richness of expert reports rather than flattening them into binary vulnerability indicators (Zhang et al., 5 Jun 2026).

A common misconception is that an auditing dataset is interchangeable with a vulnerability detection benchmark. GiANT rejects that equivalence. Its target artifact is multi-dimensional by construction, so it is intended to support not only detection, but also summarization, mitigation recommendation, and gas optimization (Zhang et al., 5 Jun 2026).

3. Framework architecture

GiANT uses a three-phase pipeline: Structured Data Generation, Iterative Semantic Refinement, and Code Mapping and Scoring Oracle (Zhang et al., 5 Jun 2026).

In the first phase, the system crawls publicly available Code4rena reports from 2021 to 2025. Because the goal is to pair findings with source code, reports without publicly accessible repositories are filtered out. Since the reports appear in heterogeneous formats, PDFs are converted into Markdown using PyMuPDF. A regex-based hierarchical parser then exploits the report structure by segmenting documents into severity-based chunks such as High Risk Findings, Medium Risk Findings, Low Risk Findings, Non-Critical Findings, and Gas Optimizations. For each chunk, it isolates individual findings, extracts metadata such as finding id and issue URL, and retains the remainder as a local content block (Zhang et al., 5 Jun 2026).

The extraction step itself is described as LLM-based Context-Augmented Semantic Extraction. It uses local finding text together with external discussion from GitHub issues referenced by the issue URL. The prompting strategy explicitly uses In-Context Learning (ICL) and Chain-of-Thought (CoT) to populate a JSON schema whose target fields include vulnerable code, description, impact, proof of concept, and mitigation. An Integrity Validator immediately checks whether the resulting JSON record is missing fields or is too brief or invalid; deficient records are re-extracted (Zhang et al., 5 Jun 2026).

The second phase, Iterative Semantic Refinement, is designed to improve fidelity and suppress hallucination. A Logical Consistency Validator compares the source report and extracted JSON to identify omissions, records discrepancies in a Quality Assurance (QA) report, and triggers Knowledge Recovery to re-extract missing fragments. Hallucination control is implemented through a text-inclusion algorithm that measures lexical overlap between extracted output $y$ and source text $x$. The paper states that records with

$\text{overlap}(y, x) < 0.6$

are discarded as hallucination risk (Zhang et al., 5 Jun 2026).

The principal QA mechanism is an LLM-as-a-Judge component. It evaluates extracted records on a 5-point Likert scale using three criteria: Accuracy, Completeness, and No Summarization. The judge returns a PASS/FAIL verdict and a diagnostic critique. Failed records enter a recursive repair loop in which a knowledge rescuer synthesizes local report context and external issue discussion, the extractor reruns with rejection history, and the judge reevaluates the revised record. After up to 3 retries, unrecoverable records are discarded. The paper summarizes this repair cycle with notation of the form

$$d_{\text{new}} \leftarrow \text{LLM\_Extract}(C, r, H),$$

$$(v, \rho) \leftarrow \text{LLM\_Judge}(d_{\text{new}}, C),$$

where $v \in \{\text{PASS, FAIL}\}$ and $\rho$ is the critique (Zhang et al., 5 Jun 2026).

The third phase addresses source-code grounding. Because extracted vulnerable-code snippets are often incomplete or linked unstably, GiANT applies a dual-path mapping strategy. It performs precise mapping when reports contain explicit GitHub anchors such as #L100-L110, and inference-based mapping otherwise, with the LLM resolving target filenames from vulnerability context, searching within relevant files, and falling back to global search if necessary. Directories such as test, mock, and node_modules are filtered out. A subsequent LLM-based Alignment Discriminator scores semantic consistency between retrieved code and the documented description and impact on a 5-point Likert scale; only records with score $\ge 4$ are retained (Zhang et al., 5 Jun 2026).

4. Corpus scale and composition

The evaluated corpus is drawn from 388 real-world audit reports and represents 340 distinct projects. The final GiAnt Corpus contains 7,711 findings distributed across five severity categories (Zhang et al., 5 Jun 2026).

Severity category	Findings
High Risk	1,048
Medium Risk	2,443
Low Risk	782
Non-Critical	723
Gas Optimization	2,715

This composition is significant because it makes the corpus simultaneously a security dataset and an efficiency dataset. Gas optimization accounts for 2,715 records, so the corpus is not restricted to exploit-oriented failure modes. The paper explicitly presents this as part of the corpus’s broader usefulness for smart contract engineering (Zhang et al., 5 Jun 2026).

The construction cost is also reported. GiANT consumed about \$597.2 USD, 21.3 million tokens, and an average of 62.6k tokens per report. These values situate the framework between purely manual curation and unconstrained large-scale LLM processing: the system is automated, but its extraction, QA, and recursive repair stages remain computationally nontrivial (Zhang et al., 5 Jun 2026).

5. Quality control and empirical reliability

The paper’s central empirical claim is that the extraction pipeline achieves high fidelity relative to human judgment. Manual evaluation was conducted on 400 sampled records stratified across the five severity categories. The first two authors manually reviewed them; 350 were independently split for assessment, and 50 were jointly reviewed for inter-coder agreement (Zhang et al., 5 Jun 2026).

The reported outcome is a mean quality score of $\kappa = 0.88$0 out of 5 and inter-rater agreement $\kappa = 0.88$1. The paper characterizes this as near-perfect agreement. It also invokes a Cochran-style finite population argument to justify the sample size, claiming approximately 95% confidence with a margin of error around 4.75% (Zhang et al., 5 Jun 2026).

The evaluation protocol reflects the framework’s extractive rather than abstractive orientation. The judge criteria include No Summarization, meaning that the desired output is not a compressed paraphrase of the source report, but a faithful structured preservation of expert audit content. This is an important methodological point because many LLM pipelines optimize for concise reformulation; GiANT instead prioritizes information retention and traceability to source text (Zhang et al., 5 Jun 2026).

Another misconception addressed implicitly by the results is that lexical or semantic similarity alone suffices to validate extraction. GiANT combines overlap filtering, logical consistency validation, recursive repair, and LLM judging. This suggests that no single QA signal is treated as adequate in isolation; reliability is produced by layered validation rather than by one metric (Zhang et al., 5 Jun 2026).

6. Benchmark tasks, observed model limits, and research significance

To demonstrate the corpus’s practical utility, the paper benchmarks four state-of-the-art LLMs: GPT-3.5-turbo, GPT-4o, Qwen3-Coder-480B-A35B-Instruct, and DeepSeek-V3. These models are evaluated on vulnerability detection, code summarization, mitigation recommendation, and automated gas optimization (Zhang et al., 5 Jun 2026).

For vulnerability detection, the tasks are framed as binary and multi-class classification, evaluated with Accuracy, Precision, Recall, and F1. The paper reports that the models perform reasonably in binary detection, but that performance collapses on severity classification, with multi-class F1 scores only about 0.280–0.302. The result is significant because severity assignment is one of the expert judgments that many existing datasets either omit or oversimplify (Zhang et al., 5 Jun 2026).

For code summarization, models generate natural-language explanations of vulnerable code and perform risk or impact analysis. The evaluation uses BERTScore and an LLM-judge Likert score. The paper reports relatively high BERTScore values but low expert-like judge scores, indicating that semantic resemblance does not necessarily imply faithful capture of audit logic (Zhang et al., 5 Jun 2026).

For mitigation recommendation, the same metrics are used. This is reported as the strongest task for the evaluated models: all achieve high semantic similarity and relatively strong judge scores, with GPT-4o, Qwen3-Coder, and DeepSeek-V3 around the 4.0 range. Even here, however, output variance remains high (Zhang et al., 5 Jun 2026).

For gas optimization, the models refactor code while attempting to preserve functionality and improve gas efficiency. The paper concludes that LLMs show a foundational ability on this task, but that quality is unstable and highly variable. Across tasks, the paper’s synthesized conclusion is that zero-shot prompting is not sufficient for professional smart contract auditing (Zhang et al., 5 Jun 2026).

Taken together, these benchmarks position the GiAnt Corpus as both a dataset and a diagnostic instrument. It is large enough and structured enough to expose a gap between plausible language generation and professional auditing competence. A plausible implication is that future progress will depend less on generic zero-shot prompting and more on fine-tuning, retrieval augmentation, and task-specific reasoning systems, which the paper identifies as natural next steps (Zhang et al., 5 Jun 2026).