Statistical significance of observed performance differences

Ascertain whether the observed performance differences between VulGNN and the baseline detectors (including ReVeal, RoBERTa, CodeBERT, GraphCodeBERT, GPT-2 Base, CodeGPT, PolyCoder, T5 Base, CodeT5 Small/Base, and NatGen) are statistically significant rather than due to chance by applying appropriate significance tests to precision, recall, F1-score, accuracy, and false positive rate results under the DiverseVul evaluation setups (“Train on Prev+Diverse, Test on Unseen Projects” and “Train and Test on Prev+Diverse with no overlaps”).

Background

The study evaluates VulGNN against a range of language-model and GNN baselines on DiverseVul using multiple metrics (precision, recall, F1-score, accuracy, and false positive rate) under both in-distribution and unseen-project configurations. Although results are reported, no formal statistical tests are conducted to determine whether performance differences are statistically significant.

The authors explicitly note the absence of significance testing, leaving unresolved whether the observed performance gaps reflect genuine model differences or sampling variability. Establishing statistical significance would strengthen the evidence supporting comparative claims in real-world vulnerability detection contexts.

References

Finally, since we did not perform statistical tests, we cannot formally assess whether observed differences between models are significant or due to chance.

— Software Vulnerability Detection Using a Lightweight Graph Neural Network  (2603.29216 - Farmer et al., 31 Mar 2026) in Section 5.2.4, Conclusion Threats to Validity (Discussion)

Our seed count is small: three seeds suffice for the exact test on the headline last-place ranking ($p \approx 0.037$; conservative variants 0.074 and 0.111) but not for tight variance estimates, so the seed-44 inversion between the two F1 shapes is left open rather than resolved.

— Reinforcement Learning with Verifiable Rewards for Small Search Agents  (2609.28765 - Jayadas et al., 23 Sep 2026) in Section 6, “Limitations and Future Work”

We present MultiAttenGastro not as a demonstrated best-in-class attention module --- our own multi-seed comparison could not distinguish it with statistical confidence from CBAM or Coordinate Attention on the one dataset where it performs best (Sect.~\ref{sec:attn-comparison}) but as a concrete case study and a diagnostic protocol (multi-seed ablation, CKA, significance testing) for identifying when multi-dimensional attention is likely to help GI endoscopy models, and when it is likely to add redundant capacity instead.

— MultiAttenGastro: Multi-Dimensional Attention Augmentation for Gastrointestinal Endoscopy Classification  (2609.05070 - Devarajan et al., 4 Sep 2026) in Section 7, Conclusion; see also Section 5.1.1, Comparison against Alternative Attention Mechanisms