PatchSeeker: Tracing CVEs to Fixing Commits

• PatchSeeker is a method for mapping CVE descriptions from the NVD to their corresponding vulnerability-fixing commits using LLM-generated commit messages and embedding-based retrieval.
• It employs a three-phase system—data processing, fine-tuning with contrastive learning, and FAISS retrieval—to efficiently bridge the semantic gap between vulnerability reports and code changes.
• Evaluation results show significant improvements in Recall@10 and MRR over baselines, reducing manual review effort in identifying security patches.

PatchSeeker is a method for mapping National Vulnerability Database (NVD) records to the corresponding vulnerability-fixing commits (VFCs) in open-source repositories. It addresses a specific retrieval problem in software security: NVD entries describe disclosed vulnerabilities in natural language, whereas the commits that repair them are often weakly documented, tersely named, or intentionally opaque. PatchSeeker bridges this gap by combining LLM-generated commit messages with embedding-based retrieval, so that vulnerability descriptions and commits can be compared in a shared semantic space (Nguyen et al., 9 Sep 2025).

1. Problem definition and scope

PatchSeeker targets the automated recovery of trace links between CVE descriptions in the NVD and their true VFCs. The motivation is operational as well as methodological. VFCs encode the exact code changes enacted to fix security issues, which supports vulnerability localization, patch analysis, and dataset construction. At the same time, the NVD often lacks explicit links to those commits; the reported figure is that only about 6.7% of CVEs have explicit links to their VFCs (Nguyen et al., 9 Sep 2025).

The task is difficult because the two sides of the match are expressed in different modalities and at different abstraction levels. NVD descriptions are concise vulnerability narratives, while commits frequently contain short or uninformative messages. The paper characterizes this as a semantic gap between natural-language reports and low-level code changes. PatchSeeker is designed specifically for that gap: it does not attempt to classify whether a commit is security-relevant in general, nor to test whether a patch is present in a compiled binary. Its task is narrower and more structured—rank candidate commits for a disclosed vulnerability record.

This problem formulation places PatchSeeker within security patch tracing rather than generic patch mining. In contrast to systems that search over repository history using lexical cues alone, PatchSeeker explicitly uses message synthesis and representation learning to model the correlation between vulnerability descriptions and candidate fixing commits.

2. System architecture

PatchSeeker is described as a three-phase system: data processing, fine-tuning, and retrieval (Nguyen et al., 9 Sep 2025).

In the data-processing phase, the developer-written commit message is extracted for each candidate commit. If the original message is short or lacks detail, PatchSeeker uses the CCT5 model to generate a descriptive message from the commit diff. The generated message and the original message are then combined into a richer textual representation. The paper states that these generated messages act as a semantic bridge, effectively closing the information gap between natural language reports and low-level code changes.

In the fine-tuning phase, PatchSeeker employs a bi-encoder architecture built on the Qwen-3 LLM. The model is trained with contrastive learning so that the embedding of a CVE description is close to the embedding of its true fixing commit and separated from non-matching commits. The training objective is the InfoNCE loss:

$$\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(e_d, e_{c_k}) / \tau)}{\sum_{i=1}^{n} \exp(\text{sim}(e_d, e_{c_i}) / \tau)}$$

where $e_d$ is the embedding of the CVE description, $e_{c_k}$ is the embedding of the true VFC, $\text{sim}(\cdot,\cdot)$ is cosine similarity, and $\tau$ is a temperature parameter.

In the retrieval phase, a new CVE description and the candidate commits are embedded with the trained model, and FAISS is used to rank the candidates by proximity in embedding space. This yields a top-$k$ retrieval pipeline rather than a binary classification pipeline. The design choice is important because the operational goal is usually to minimize analyst review effort while still surfacing the true VFC near the top of the ranking.

3. LLM-generated commit messages as the central mechanism

The distinctive mechanism in PatchSeeker is not merely the use of embeddings, but the use of LLM-generated commit messages to enrich poor repository metadata (Nguyen et al., 9 Sep 2025).

The paper assigns two roles to LLMs. First, the CCT5 model is used for commit message synthesis from code diffs. This is intended for commits whose original messages are vague, minimal, or otherwise semantically weak. Second, the Qwen-3 bi-encoder is fine-tuned to embed both CVE descriptions and enhanced commit messages into a shared vector space. The method therefore does not rely on raw commit text alone; it deliberately reconstructs a higher-information textual proxy for the code change.

This design directly addresses a recurrent difficulty in security patch analysis: silent fixes and weak commit documentation. Related work on security patch identification has likewise emphasized that many security-relevant commits are not explicitly labeled and may be difficult to recognize from repository metadata alone (Sawadogo et al., 2020). SPI similarly treats commit messages and code revisions as complementary signals for identifying security patches, though its task is security-patch classification rather than CVE-to-VFC tracing (Zhou et al., 2021). PatchSeeker differs in using generated messages not as an explanation layer but as a retrieval primitive.

A plausible implication is that PatchSeeker benefits most in precisely the cases where commit messages are least informative. That implication is supported by the ablation results: adding CCT5-generated messages for terse commits improved Recall@1 by 14.7%, and the paper reports that the selection of backbone LLMs also makes a positive contribution.

4. Training data, evaluation protocol, and results

The benchmark evaluation uses 5,000 CVEs from 2,070 repositories, with realistic negative sampling defined as all commits in the year before disclosure. During fine-tuning, hard negatives are sampled from the top results of BM25 and CoCondenser retrievers. The reported metrics are Recall@K, Mean Reciprocal Rank (MRR), Manual Effort@K, and Precision@K for the implicit set (Nguyen et al., 9 Sep 2025).

The main benchmark results are summarized below.

Evaluation setting	Metric	Result
Benchmark dataset	Recall@10	0.871
Benchmark dataset	MRR	0.739
Benchmark dataset	Manual Effort@10	50.8% reduction vs Prospector
Recent CVEs, explicit-set	Recall@10	0.786
Recent CVEs, implicit-set	Precision@10	0.71

On the benchmark dataset, PatchSeeker achieves 59.3% higher MRR and 27.9% higher Recall@10 than the best-performing baseline, Prospector. The paper also reports a reduction of Manual Effort@10 by 50.8% relative to Prospector. On recent CVEs, PatchSeeker reaches Recall@10 $= 0.786$ on the explicit-set, reported as 27% higher than the strongest baseline, and Precision@10 $= 0.71$ on the implicit-set, reported as a 97% improvement over the best baseline.

The ablation study is central to the interpretation of these numbers. It shows that both the commit message generation method and the selection of backbone LLMs contribute positively. Qwen-3 significantly outperformed prior code-specific and smaller LLMs, including CodeReviewer and Llama 2/Llama 3, and the maximum difference in Recall@1 between the best and worst backbone is reported as about 45%. The qualitative analysis further indicates that PatchSeeker can recover matches even when there is no direct keyword overlap between the CVE description and the true fixing commit.

5. Relationship to adjacent patch-analysis problems

PatchSeeker belongs to a broader family of research on security patches, but its objective is distinct from several neighboring tasks.

Patch tracing for disclosed vulnerabilities is the closest neighboring task. PatchFinder is a two-phase framework that also links CVEs to patch commits, using initial retrieval followed by supervised re-ranking. It reports Recall@10 of 80.63% and MRR of 0.7951 on 4,789 CVEs from 532 OSS projects (Li et al., 2024). PatchSeeker addresses the same high-level problem, but emphasizes LLM-generated commit message augmentation as the semantic bridge between CVE descriptions and commits.

Security patch identification is related but not identical. “Learning to Catch Security Patches” formulates the problem as automatic identification of security-relevant patches in repositories and uses a Co-Training-based approach over text and code views, reporting precision over 90% and recall over 80% (Sawadogo et al., 2020). SPI likewise identifies security patches from commits using a commit-message neural network and a code-revision neural network, achieving F1-score as high as 87.93% and precision of 86.24% (Zhou et al., 2021). These systems answer whether a commit is likely security-relevant; PatchSeeker instead ranks which commit fixes a specific disclosed CVE.

Patch presence testing is different again. PS$^3$ tests whether a known patch is present in a target binary by comparing semantic symbolic signatures, achieving precision $0.82$, recall $e_d$0, and F1 $e_d$1 on 3,631 $e_d$2 pairs (Zhan et al., 2023). Lares performs LLM-driven code slice semantic search for patch presence testing and reports average F1 $e_d$3 across optimization settings while eliminating the need for compilation (Li et al., 3 Nov 2025). Those approaches operate after a patch has already been identified; PatchSeeker operates earlier, at the stage of discovering the fixing commit itself.

This division of labor suggests a pipeline interpretation: a CVE-to-VFC tracer such as PatchSeeker can supply the fixing commit, while patch presence testing systems can subsequently determine whether that fix exists in deployed binaries. That interpretation is inferential, but it is consistent with the task boundaries stated in the respective papers.

6. Limitations, failure modes, and open questions

PatchSeeker discusses several limitations explicitly (Nguyen et al., 9 Sep 2025). One is tokenization sensitivity: failure cases can arise when tokenization breaks critical terms and disrupts semantic matching. The paper gives the example of splitting “multiframe” such that the “frame” clue is lost. Another limitation is the quality of the ground truth. The MoreFixes dataset is described as not error-free, and implicit cases—those without known VFCs in the NVD—can include unfixed or poorly documented vulnerabilities.

A further limitation concerns commit message augmentation. At present, the generated and original messages are simply concatenated. The paper identifies better integration or structuring of those signals as a future direction. Evaluation bias is also acknowledged: even with 2,070 repositories and 5,000 CVEs, external validity is not fully assured.

The future directions named in the paper remain within the same problem frame. They include deeper analysis of which linguistic cues in commit messages optimize VFC match performance, experimentation with aggregation methods beyond concatenation, and expansion of benchmarks to more diverse and implicit vulnerability scenarios. More broadly, PatchSeeker indicates a shift in security patch tracing from handcrafted or primarily lexical matching toward representation learning that uses synthetic intermediate descriptions. Whether that shift generalizes across ecosystems, commit styles, and disclosure practices remains an open research question, but the reported results establish the viability of LLM-generated commit descriptions as an effective retrieval signal.