Quantify taxonomy-depth bias in reference-based evaluation

Quantify the potential evaluation bias caused by using three-level taxonomies for both ground-truth annotations and SHIP's structured representations in reference-based comparisons.

Background

The paper evaluates SHIP primarily through its ability to answer specified analytical workloads and notes that the ground-truth annotations and SHIP outputs both use three-level taxonomies. This shared design choice may advantage SHIP in reference-based comparisons by aligning the evaluation structure with the representation produced by the method.

The authors explicitly state that this potential bias has not been quantified. The unresolved problem is therefore to measure how evaluation results change under alternative taxonomy depths and reference schemas, thereby determining whether the reported comparisons remain robust.