Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
173 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

UntrustVul: An Automated Approach for Identifying Untrustworthy Alerts in Vulnerability Detection Models (2503.14852v1)

Published 19 Mar 2025 in cs.SE

Abstract: Machine learning (ML) has shown promise in detecting vulnerabilities. To review vulnerabilities detected by ML predictions, developers manually assess suspicious lines in their interpretations. However, studies have revealed that these models often learn and predict based on irrelevant features frequently appearing in vulnerable code. This leads to predictions that may correctly flag vulnerable functions but for the wrong reasons, which we call untrustworthy. These predictions can mislead developers, hindering them from locating the vulnerabilities. This increases the efforts of manual assessment and, worse, risks creating flawed patches that fail to address existing vulnerabilities and even introduce new ones. Hence, automated approaches are needed to detect untrustworthy predictions, preventing overlooked vulnerabilities and alleviating the burden of manual assessment. We propose UntrustVul, the first automated approach to identify untrustworthy vulnerability predictions. Given a vulnerability prediction during inference, UntrustVul systematically assesses whether suspicious lines annotated by the prediction are vulnerability-unrelated. It simulates developers' rationales, considering a line unrelated if (1) it is absent from historical vulnerabilities and (2) it cannot reach any vulnerabilities in execution flows. UntrustVul assesses (1) by analysing its syntactic meaning using deep representations to determine whether it is syntax-benign. To assess (2), UntrustVul traces dependencies of the syntax-benign lines on other suspicious lines using static and rule-based analyses. We evaluate UntrustVul on 155K vulnerability predictions by four models across three datasets. UntrustVul effectively detects untrustworthy predictions with an F1-score of 82%-94% and helps improve the ability of models to detect vulnerabilities by up to 321% in F1-score and 100% in trustworthiness.

Summary

We haven't generated a summary for this paper yet.