Ground-Truth Evaluation (GTEval)

• Ground-Truth Evaluation (GTEval) is a methodology that defines and uses precise reference data to assess the accuracy of analysis tools.
• It integrates multiple sources such as symbol tables, debug data, and heuristic disassembler outputs to mitigate common pitfalls and biases.
• It underpins robust binary analysis by employing metrics like precision, recall, and F1 score to guide reproducible tool comparisons.

Ground-Truth Evaluation (GTEval) encompasses the rigorous methodologies, protocols, and metrics required to assess the correctness and comparative quality of analysis tools, models, and algorithms against a reference set of “ground truth”—the authoritative set of answers, labels, or annotations. Ground-truth evaluation is foundational for domains ranging from binary analysis to clustering and computer vision, but is fraught with context-dependent definitions, challenges in ground-truth construction, and pitfalls in interpretation, especially as tasks and datasets scale or shift to machine-learning–centric regimes.

1. Definitions and Scope of Ground Truth

The core of GTEval is the formal identification of what constitutes “ground truth” in a given evaluation context. In binary analysis, ground truth is understood as the set of “correct answers” (e.g., function entry points, boundaries, instruction classifications) to which a tool’s output is compared. Critically, ground truth is context-dependent and must be specified along multiple axes:

• Abstraction Level: Is the ground truth defined at the instruction level (instruction vs. data), the function level (entry points, boundaries), the semantic/source mapping, or higher-level constructs such as control-flow graphs or variable mappings?
• Model Orientation: Binary-centric ground truth regards the compiled artifact solely, while source-centric ground truth seeks to preserve relationships to the original source constructs.
• Temporal Reference: Compile-time ground truth (symbol tables, DWARF, compiler IR) includes all statically present code, including unreachable or unresolved artifacts. Run-time/trace-based models limit ground truth to instructions executed during a particular trace (Alves-Foss et al., 2022).

This necessity of precise scope is central: any misalignment between a tool’s evaluated output and the ground truth abstraction/model fundamentally biases the metric results.

2. Sources of Ground Truth and Their Pitfalls

GTEval requires the assembly of ground-truth datasets through a combination of metadata extraction, manual annotation, or automated program analysis. Each source class introduces unique strengths and liabilities:

Source	Advantages	Disadvantages
Symbol tables	Unstripped: names, entry points, sizes	Padding, symbol aliasing, optimizations
Debug data (DWARF/PDB)	Parameter, variable, and range metadata	Omitted/mis-emitted tags; parsing errors
Disassembler heuristics	No source required, heuristic CFG recovery	1–19% error on function starts; aliasing
Compiler IR/instrument.	“Oracle” mapping, byte-exact labeling	Compiler/version specific, incomplete
Manual annotation	Highest confidence if careful	Not scalable, error-prone
Dynamic instrumentation	Precise executed code labeling	No static coverage, input/schedule dep.

Failure to correctly account for these weaknesses remains a leading cause of misinterpreted evaluation, particularly in the presence of compiler optimizations, aliasing, or tool-generated code (Alves-Foss et al., 2022).

3. Methodological Workflows and Best Practices

Robust ground-truth evaluation mandates a workflow that ensures consistency, reproducibility, and explainability:

1. Explicitly Define Evaluation Scope: State and document the model, abstraction level, relaxations (e.g., inclusion/exclusion of inlined code), and the function/entity definition before any extraction or evaluation.
2. Combine and Cross-Validate Multiple Sources: Integrate symbol tables, debug info, and heuristic disassembler results for maximal coverage and verification; cross-check for internal consistency.
3. Prune or Repair Ambiguous Binaries: Exclude any binary where ground-truth extraction yields inconsistencies (e.g., missing debug sections, stripped symbols).
4. Automate Exhaustive Assertion Checking:
   • Each code byte classified as instruction or data.
   • Symbol and debug-range matches for functions.
   • Non-overlapping function address ranges.
   • Sum of all function/padding lengths equals total section size.
5. Document All Extraction Scripts and Workflow: Scripts, command parameters, and output logs must be published alongside the dataset to allow external reproducibility.
6. Explicitly Annotate Special Cases: Label non-returning/library functions accurately for correct control-flow graph construction and downstream analysis (Alves-Foss et al., 2022).

A release checklist for ground-truth datasets includes: source repository, build scripts with locked compiler versions, raw and symbol data, full ground-truth target files, and validation reports.

4. Evaluation Metrics and Coverage Measures

GTEval universally employs information retrieval metrics, instantiated on the problem domain:

• Precision (P):

$P = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FP}}$

• Example: Number of function-entry addresses correctly recovered over total reported.
  • Recall (R):

$R = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN}}$

• Example: Ground-truth function entries detected over total present.
  • F₁ Score:

$F_1 = 2\frac{P \cdot R}{P+R}$

• Coverage: For instruction decoding,

$$C_{\mathrm{inst}} = \frac{|\text{decoded}_\text{instructions} \cap \text{GT}_\text{instructions}|}{|\text{GT}_\text{instructions}|}$$

Fundamental for binary analysis, boundary-precision and boundary-recall examine the accuracy of byte-range assignment, not just entry address points. Ground-truth evaluation may require additional, task-specific measures such as set-similarity, Jaccard indices, or structural graph matching in other application domains (Alves-Foss et al., 2022).

5. Implications for Learning-Based Analysis

Machine learning models are acutely sensitive to ground-truth quality:

• Any ambiguity or systematic error in the ground-truth dataset (e.g., heuristics from disassembler output or incomplete metainformation) is internalized, potentially causing persistent model blind spots.
• Training or evaluating on non-representative ground-truth (e.g., single-compiler, one optimization level, unchecked heuristic output) impairs generalization and propagates existing tool weaknesses.
• Robust ML-based binary analysis requires highly consistent compiler-instrumented or manually validated ground-truth and broad diversity in binary origin for meaningful model comparison and development.
• Full documentation of extraction and labeling process is essential to expose hidden dataset biases (Alves-Foss et al., 2022).

6. Addressing Common Pitfalls and Misconceptions

Significant methodological errors continue to propagate in binary-analysis benchmarking:

• Equating Symbol Table with Ground Truth: Symbol tables may include aliasing, padding, and compiler-generated artifacts and do not provide complete functional mapping.
• Over-Reliance on Commercial Disassembler Outputs: Heuristic errors, especially on indirect jumps and large NOP sequences, invalidate the use of such outputs as an “oracle” for function boundaries or control-flow.
• Neglecting Compiler-Induced Artifacts: Inlining, tail calls, exception stubs, or runtime support routines inserted by the compiler/linker may not correspond to source-level entities but are present in the binary.
• Confusion around Code Versus Data: The location (.text, .rodata, etc.) and execution status (executed, dynamically copied) of code/data must be distinguished carefully.
• Artificially Inflating Evaluation Metrics: Scripts that “hack” around rare/short function corner-cases or ambiguities undermine reproducibility and fairness of benchmarking.

Systematic adherence to rigorously defined ground-truth models, multi-source validation, and reproducible metric computation is recommended to avoid these pitfalls (Alves-Foss et al., 2022).

7. Toward Standardization and Community Practice

GTEval in binary analysis is moving toward systematic, explicit, and reproducible protocols, with the following recommendations:

• Every evaluation must specify the definition and scope of ground truth before extraction.
• Published datasets must include all source code, extraction scripts, and validation artifacts.
• Results must be computable by IR-style metrics at task-appropriate levels (function, instruction, CFG).
• Comparability across tools and research works requires explicit reporting of all ground-truth modeling assumptions and extraction procedures.
• Community-accepted protocols, including automated consistency checks and open-source extraction code, are essential to achieving robust, interpretable, and fair comparisons (Alves-Foss et al., 2022).

By embracing methodological transparency and context-aware definition, the binary analysis community can ensure that ground-truth evaluation both supports reliable tool comparison and enables sound ML-driven research.