Audit the provenance and reliability of training annotations

Determine the inter-annotator agreement statistics, labeling guidelines, and error rates for the Roboflow-sourced rice detection annotations used to train the AgriNav detection models, thereby establishing whether quantitative accuracy claims are reliable.

Background

The AgriNav detection models are trained on rice detection datasets obtained from Roboflow. Although the datasets are described as curated and human-labeled, the research team did not author or independently audit the annotations. Consequently, the quality and consistency of the labels—particularly the inter-annotator agreement, annotation rules, and error rates—have not been established.

Because detection performance is sensitive to annotation granularity, bounding-box conventions, and labeling errors, the paper identifies a targeted audit of a stratified sample as necessary before making quantitative claims about model accuracy. This unresolved annotation-quality issue is distinct from the model-training and cross-dataset evaluation limitations described elsewhere in the paper.].

quote].