CAPYBARA: Dataset for Decompiled Function Summaries

• CAPYBARA is a dataset of decompiled pseudo-C functions paired with concise documentation summaries, specifically designed for reverse engineering tasks.
• It is built using the BinSwarm pipeline from 1,700 C projects, featuring functions across various compiler optimization levels and both stripped and unstripped binaries.
• Evaluation with a fine-tuned CodeT5 model, BinT5, showed a state-of-the-art BLEU-4 of 58.82 on decompiled code, highlighting challenges like identifier loss and decompilation artifacts.

Searching arXiv for the specified paper to ground the article in the publication record. CAPYBARA is a dataset for summarising decompiled binary functions, introduced in the context of extending source-code pre-trained LLMs to reverse-engineering workflows. It consists of aligned decompiled function bodies and source-level documentation summaries derived from C projects, with coverage across compiler optimisation levels and both stripped and unstripped binaries. In the associated study, CAPYBARA is paired with a fine-tuned CodeT5-base model, BinT5, to evaluate whether source-code LLMs can transfer to decompiled pseudo-C; the reported results include BLEU-4 scores of 58.82 on decompiled code and 44.21 on synthetically stripped decompiled code, indicating substantial but incomplete transfer to lower-signal binary-derived inputs (Al-Kaswan et al., 2023).

1. Scope and research objective

CAPYBARA was constructed to address a specific gap in automated binary analysis: while reverse engineering often requires understanding programs whose source code is unavailable, existing program summarisation work had mainly focused on source code rather than decompiled binaries. The dataset therefore targets function-level semantic labelling for decompiled pseudo-C, a task motivated by the fact that reverse engineering is time-consuming and that much of this effort is spent assigning semantic information to functions (Al-Kaswan et al., 2023).

The dataset is organised around decompiled function–documentation pairs rather than raw binaries alone. This design reflects the paper’s core task formulation: mapping a decompiled function body to a concise natural-language summary extracted from source-level comments. A plausible implication is that CAPYBARA is intended less as a general binary-analysis corpus than as a supervised benchmark for semantic recovery from decompilation artifacts, especially under varying levels of symbol information and compiler transformation.

A common misconception would be to treat CAPYBARA as a conventional source-code summarisation dataset with a binary flavour. The construction procedure instead centres decompiled pseudo-C and stripped variants, making it specifically relevant to reverse engineering, binary comprehension, and downstream settings in which source-level identifiers or comments are unavailable.

2. Data acquisition and alignment pipeline

The original data source is the BinSwarm pipeline from FUNCRE, which harvested approximately 1,700 C/GCC projects from GitHub using Travis-CI and Ubuntu targets (Al-Kaswan et al., 2023). Each project was built in Docker at four compiler-optimisation levels, namely -O0, -O1, -O2, and -O3, and then stripped with GNU strip. Both stripped and unstripped binaries were subsequently decompiled with Ghidra’s pseudo-C decompiler.

BinSwarm produces aligned triples of the form $$\langle \text{source-C}, \text{decompiled-C}, \text{stripped-decompiled-C} \rangle$$ by matching function names in the symbol table. Because this matching signal is absent after stripping, the alignment procedure depends critically on symbol-table availability during data generation. This suggests that the stripped setting is not merely noisier but structurally lower-signal, since the principal alignment mechanism used upstream no longer exists in the deployed artifact.

Preprocessing focused on extracting each function’s concrete pseudo-C body together with its source-level documentation comments. The authors used srcML on the original .c and .h files to locate the comment block immediately preceding each function signature. They also stripped compiler-inserted macros, standardised whitespace, and truncated functions longer than 512 tokens at training time. These steps define a normalised function-level representation suitable for seq2seq fine-tuning while preserving the decompilation-specific lexical and structural artifacts that distinguish the task from source summarisation.

3. Summary extraction and corpus curation

CAPYBARA does not use arbitrary comments as supervision; it applies explicit heuristics to isolate a one-sentence summary from a multi-line C comment block. The extraction rules are cascading. If an @brief or @purpose tag exists, the first sentence after it is taken. Otherwise, if a Description: line is present, its first sentence is selected. Otherwise, if @param or @v tags exist, the sentence immediately before the first such tag is taken. If none of these cases applies and the comment is single-line, that line is used (Al-Kaswan et al., 2023).

This rule set is important because it defines the target variable more narrowly than generic documentation generation. The task is not to reproduce full docstrings or arbitrary prose, but to predict a compressed summary sentence derived from documentation conventions in C projects. A plausible implication is that performance metrics reflect both semantic recovery from decompiled code and the regularity of comment-writing conventions embedded in the corpus.

The authors further filtered examples whose extracted summary was shorter than 3 tokens or longer than 256 tokens, contained URLs, HTML, file paths, or developer notes such as FIXME and TODO, or was identified as non-English with FastText language identification. These exclusion criteria reduce noise and constrain the output space to concise English summaries. They also mean that CAPYBARA is not a full-spectrum archive of binary-adjacent documentation, but a curated benchmark designed for learnable function summarisation.

4. Dataset composition, distributions, and metadata

The overall corpus contains 214,587 aligned $$\langle \text{decompiled-C}, \text{summary} \rangle$$ pairs from unstripped binaries, plus 14,245 from stripped binaries. After deduplication, the sizes are 79,673 for decompiled data and 7,826 for stripped data (Al-Kaswan et al., 2023).

The optimisation-level distribution for unstripped decompiled functions is uneven: approximately 6,000 functions at -O0, approximately 40,000 at -O1, approximately 120,000 at -O2, and approximately 48,000 at -O3. Thus, -O2 constitutes the majority of the corpus. Function-size statistics also show a marked expansion under decompilation. Source-C bodies have mean length approximately 399 tokens with $\sigma \approx 300$ and mean length approximately 30.8 lines, whereas decompiled bodies have mean length approximately 779 tokens with $\sigma \approx 600$ and mean length approximately 53.4 lines. Documentation-length distributions differ sharply between multi-line and single-line summaries: the former average approximately 42.6 tokens, the latter approximately 8.1 tokens.

Aspect	Value
Unstripped aligned pairs	214,587
Stripped aligned pairs	14,245
Deduplicated decompiled pairs	79,673
Deduplicated stripped pairs	7,826
Dominant optimisation level	-O2 at approximately 120,000 functions

The stored metadata include project ID, compiler, exact -Ox flag, and a tag distinguishing stripped, unstripped, and “demi-stripped” examples. The train/validation/test split is cross-project with an 80/10/10 partition. This cross-project design matters methodologically because it reduces the likelihood that evaluation primarily measures project-local lexical repetition rather than transfer across repositories. The presence of compiler and optimisation metadata also makes CAPYBARA suitable for controlled analyses of compilation effects on summarisation.

5. Synthetic variants and deduplication methodology

A central extension of CAPYBARA is the “demi-stripped” synthetic setting. From each decompiled unstripped function, the authors generated a synthetic stripped variant by replacing all identifiers—function names, parameter names, and local variables—with uniform placeholders such as FUN_0, ARG_1, and VAR_2 (Al-Kaswan et al., 2023). This preserves control-flow structure while emulating the loss of the symbol table.

The synthetic setting has a precise methodological role. It isolates the contribution of identifiers without introducing all the additional degradation present in fully stripped binaries. This suggests that CAPYBARA supports a graded analysis of semantic signal: unstripped decompiled code retains original identifiers, demi-stripped code removes them synthetically while preserving other decompilation characteristics, and fully stripped code reflects the more severe information loss encountered in practice.

Near-duplicate removal was performed with a fork of the SERG near-duplicate-code-remover from Allamanis et al. 2020, which clusters functions using n-gram overlap and light edit-distance features. Using the standard configuration, the procedure compared every pair of functions, formed clusters of near-duplicates such as library stubs or auto-generated code, and randomly retained one member per cluster while dropping the rest. The resulting deduplication rates were approximately 63% for decompiled data, reducing 214K to 79K, and approximately 45% for stripped data, reducing 14K to 7.8K. These large reductions indicate that duplicate and near-duplicate content is a substantial property of the raw corpus rather than a marginal artifact.

6. Model adaptation, evaluation, and reported results

The associated model, BinT5, is obtained by fine-tuning CodeT5-base, a 220M-parameter model pretrained on mixed PL/NL tasks across eight languages including C and C++. Inputs are tokenised with maximum length 512 to accommodate longer decompiled functions, and fine-tuning is performed on the decompiled-to-summary task with batch_size=2 and grad_accum=24, yielding an effective batch size of 48 (Al-Kaswan et al., 2023).

Evaluation includes BLEU-4, Exact Match (EM), METEOR, and ROUGE-L. The paper gives BLEU-4 as

$$\text{BLEU-4} = BP \cdot \exp \left( \frac{1}{4} \sum_{n=1}^{4} \log p_n \right),$$

where $p_n$ is the clipped n-gram precision and $BP$ is the brevity penalty, using Moses smoothing #2. The reported test-set results, including duplicates, are as follows.

Setting	BLEU-4	EM
Source-C baseline	60.83	52.19%
Decompiled (BinT5)	58.82	48.92%
Demi-stripped	44.21	35.10%
Fully stripped	11.26	1.85%

The full metric tuple is 60.83 BLEU-4, 65.33 METEOR, and 66.51 ROUGE-L for the Source-C baseline; 58.82, 63.14, and 64.51 for Decompiled (BinT5); 44.21, 47.89, and 49.59 for Demi-stripped; and 11.26, 14.50, and 17.25 for Fully stripped. The study characterises the 58.82 BLEU-4 result on decompiled code as state of the art. The gap between decompiled and fully stripped performance, together with the much stronger demi-stripped result relative to fully stripped, supports the paper’s conclusion that identifiers are critical but not sufficient: decompilation artifacts such as missing control flow and mis-typed variables remain major obstacles.

Ablation results further refine this picture. For decompiled code, BLEU-4 by optimisation level is 72.9 at -O0, 50.3 at -O1, 62.3 at -O2, and 54.7 at -O3. Deduplication reduces BLEU-4 by approximately 16 points on decompiled data, from 58.8 to 42.5, and by approximately 19 points on demi-stripped data, from 44.2 to 25.4. Halving or quartering the training set reduces BLEU by only a few points, which the authors interpret as evidence of strong transfer from CodeT5 pretraining. Taken together, these results indicate that apparent performance can depend heavily on duplicate structure, whereas sensitivity to raw training-set scale is comparatively limited in this setup.

7. Limitations, open problems, and downstream significance

The paper explicitly recommends further work on knowledge transfer to low-signal inputs such as stripped binaries. The demi-stripped experiments show that removing identifiers causes a major degradation, but the extremely low fully stripped performance indicates that identifier loss does not exhaust the difficulty of the problem (Al-Kaswan et al., 2023). The authors identify decompilation artifacts, including missing control flow and mis-typed variables, as the main remaining barrier.

Future directions proposed in the study include improving decompiler resilience, for example through better function-boundary detection in stripped code, and developing joint decompilation-plus-summarisation models that recover lost semantics end to end. Additional open challenges include expanding CAPYBARA to other architectures such as ARM and MIPS or to languages such as C++, performing human-in-the-loop evaluation to determine whether generated summaries measurably speed reverse-engineering tasks, adopting better semantic evaluation metrics because BLEU has known weaknesses for code, and incorporating richer API or use-pattern metadata or control-flow graphs to aid the model.

These limitations also delineate the dataset’s significance. CAPYBARA is not merely a benchmark for text generation over pseudo-C; it is an infrastructure for studying automated binary comprehension under controlled degradation of semantic cues. By making available approximately 215K function–summary pairs plus synthetic variants, together with BinT5, the work establishes a research substrate for deobfuscation, reverse engineering assistance, and cross-modal code-analysis tasks. A plausible implication is that CAPYBARA’s lasting value lies less in a single leaderboard number than in enabling systematic analysis of how much semantic information survives compilation, optimisation, stripping, and decompilation.