Characterize annotation-quality differences across semantic abstraction levels

Characterize the differences in verifier acceptance rates across application domains, algorithms, programming paradigms, and design patterns by evaluating whether function-level annotation, rather than whole-file annotation, resolves the apparent granularity mismatch between source-code evidence and Wikidata concepts.

Background

The paper reports that verifier acceptance rates decline for several semantic axes when annotations are restricted to concepts successfully linked to Wikidata. It proposes that algorithms and design patterns typically describe constructs at the level of a function or class, whereas the verification task evaluates whether the entire source file instantiates the corresponding Wikidata concept. Programming paradigms, by contrast, are more naturally file-oriented and therefore show better alignment between evidence and evaluation scope.

The proposed explanation is explicitly presented as a working hypothesis rather than an established finding. The paper leaves unresolved a more precise characterization of the phenomenon, potentially through function-level evaluation, which would test whether evaluating annotations at the granularity where the relevant construct occurs improves reliability.

References

A plausible explanation---which we adopt here as a working hypothesis to be further investigated---is that the four axes operate at different units of abstraction relative to the file: algorithm and design pattern labels typically describe constructs that live at the level of a single function or class (one method implements Dijkstra, one class implements Singleton), so when the verifier is asked whether the whole file instantiates the corresponding Wikidata concept, the granularity mismatch penalises acceptance. Paradigm is intrinsically file-oriented and benefits from natural alignment between evidence and abstraction units. A more precise characterization---possibly with function-level evaluation---is left to future work.

— CodeGraph: Open-Taxonomy Knowledge Graph for Source Code with Wikidata Grounding  (2609.29474 - Pennino et al., 24 Sep 2026) in Section 4, Quality Assurance Protocol, subsection on silver-tier re-evaluation