---
title: 'VulRTex: Automated Vulnerability Detection'
url: https://www.emergentmind.com/topics/vulrtex
type: topic
---

# VulRTex: Automated Vulnerability Detection

VulRTex is a reasoning-guided framework designed to automate the identification of vulnerability-related issue reports (IRs), particularly leveraging rich-text elements such as screenshots and code snippets. By integrating chain-of-thought (CoT) large language model (LLM) reasoning with retrieval-augmented guidance based on a database of historical vulnerability cases, VulRTex achieves state-of-the-art performance for vulnerability detection and CWE-ID classification in open-source software issue repositories, notably excelling under severe data imbalance [2509.03875].

## 1. System Architecture and Component Workflow

VulRTex orchestrates a four-stage pipeline encompassing preprocessing, reasoning database construction, case retrieval and guidance prompt generation, and LLM-based vulnerability identification.

**Preprocessing Module**
- Crawls and normalizes raw IRs, tagging rich-text modalities: `[SCR]` for screenshots, `[CODE]` for code snippets.
- Emits unified JSON records for subsequent modules.

**Vulnerability Reasoning Database Construction**
- For each historical IR, an LLM CoT agent constructs a reasoning graph $\mathcal{G}$ detailing stepwise how the vulnerability is substantiated by rich-text.
- Each step is fact-checked and augmented using external Vulnerability Awareness (VA) datasets (e.g., VDISC, OWASP), employing TF–IDF-based retrieval to inject golden facts and correct hallucinated outputs.
- Final graphs are persisted in the Reasoning Base.

**Case Retrieval Engine**
- For a new IR, each historical reasoning graph is pruned via a weighted random walk (see §3.1 for the algorithmic details) to extract the most relevant subgraph.
- Subgraphs are ranked against the target IR using TF–IDF similarity; top-$k$ cases (typically $k=3$) are selected as guides.

**LLM Reasoning Module**
- Combines retrieved subgraph narratives into a guidance prompt $P_{\mathrm{guide}}$, concatenated with a classification prompt $P_{\mathrm{identify}}$.
- The fused prompt is input to the LLM for binary vulnerability classification and CWE-ID prediction.

### VulRTex High-Level Pipeline Pseudocode

```latex
\begin{algorithm}[H]
\caption{VulRTex Pipeline}
\KwIn{Raw historical IRs $\{IR_i\}$, Target IRs $\{IR'_j\}$}
\KwOut{For each $IR'_j$: (vulnerability? label, predicted CWE-ID)}
// 1. Preprocessing
\ForEach{$IR_i$ in history}{
   Clean, tag rich-text, output JSON.
}
\ForEach{$IR'_j$ in target}{
   Same preprocessing.
}
// 2. Reasoning DB Construction
\ForEach{$IR_i$ in history}{
   $\mathcal{G}_i \leftarrow$ GenerateGraph($IR_i$)
   Reason\_Base.add($\mathcal{G}_i$)
}
// 3. Retrieval and Pruning
\ForEach{$IR'_j$ in target}{
   Retrieve top-$k$ subgraphs from Reason\_Base
}
// 4. LLM Reasoning
\ForEach{$IR'_j$}{
   $P_{\mathrm{guide}} \leftarrow$ MakeGuidancePrompt($IR'_j$, cases)
   $P_{\mathrm{id}} \leftarrow$ classification prompt
   $y_j, \mathrm{CWE}$ = LLM($P_{\mathrm{guide}} \oplus P_{\mathrm{id}}$)
}
\Return results
\end{algorithm}
```

## 2. Vulnerability Reasoning Database: CoT Graph Construction and Fact Correction

Reasoning graphs $\mathcal{G}=(\mathcal{V}, \mathcal{E})$ model the interpretive sequence for each historical IR:
- Vertices $\mathcal{V}$ correspond to observations (summaries of comments, screenshots, or code).
- Edges $\mathcal{E}$ represent actions taken by the LLM agent, selected from \{\texttt{ScrAnalyzer}, \texttt{CodeAnalyzer}, \texttt{AgentTerminator}\}.

At each step $t$, the agent operates on context $C_t$ and produces
\[
C_t = (O_1, A_1, O_2, A_2, \dots, O_t)
\]

To mitigate hallucination, each step’s context is expanded with authoritative knowledge:
\[
\mathrm{Know}_t = \{k \in \mathcal{D} : \mathrm{TFIDF}(k, \mathrm{Desc}(C_t)) > \theta_{\mathrm{sim}}\}
\]
where $\mathcal{D}$ aggregates data from KB, BigVul, OWASP, Debian, and VDISC.

A CoT prompt template is used:
> Please think step by step. For each step, select related rich-text elements (comment, [SCR], [CODE]), output an "Observation", and choose a "Tool" (Action) to proceed. Terminate when vulnerability is identified.

Paths through these graphs encode distinct reasoning chains for each IR.

## 3. Case Retrieval: Random-Walk Subgraph Pruning and Guided Prompt Formulation

Pruning each historical reasoning graph for retrieval is defined by a weighted random-walk:
- Edge weights are computed as increments in TF–IDF similarity between the endpoints and the target IR:
\[
\tilde{M}_{i,j} = \mathrm{TFIDF}(O_j, IR^{\text{tgt}}) - \mathrm{TFIDF}(O_i, IR^{\text{tgt}})
\]
- Edge sampling probability:
\[
p(O_i, O_j) = \frac{\frac{1}{\deg(O_i)} + \frac{1}{\deg(O_j)}}{\sum_{(u,v) \in \mathcal{E}} (\frac{1}{\deg(u)} + \frac{1}{\deg(v)})}
\]
- The process halts upon reaching an AgentTerminator edge; the visited nodes constitute the retrieved subgraph $\mathcal{G}_r$.

Candidate subgraphs are ranked by TF–IDF similarity to the target IR’s full text and rich-text components; top-$k$ are chosen.

Guidance prompts $P_{\mathrm{guide}}$ are constructed by concatenating path summaries from these top cases with procedural instructions for new IR analysis.

## 4. LLM-Guided Vulnerability Identification and Output Decoding

The LLM receives the prompt $P = P_{\mathrm{guide}} \oplus P_{\mathrm{identify}}$, where the identification portion is standardized:
> Please classify whether the following IR contains a vulnerability (Yes/No). If Yes, output the CWE-ID. Format: {Yes/No}, {CWE-xxx}.

The model produces both a binary label and CWE-ID. In an illustrative case (AntSword #147), the system robustly identifies XSS (CWE-79) by analogical reasoning along retrieved case paths.

## 5. Experimental Design, Evaluation, and Results

**Dataset**: 973,572 GitHub IRs (2015–2023); 4,002 vulnerability-related (39.1% with rich-text), 1,191,318 negative-class IRs. The exploitation of rich-text is crucial, as 39.1% of vulnerability IRs contain `[SCR]` or `[CODE]`.

**Split**: 60% of vul-positives for reasoning database construction (2,401 IRs), 40% for target evaluation (1,601 IRs). The negative class remains highly imbalanced.

| Split       | #IRs Total | Rich-Text | [SCR] | [CODE] |
|-------------|------------|-----------|-------|--------|
| Vul (hist)  | 2,401      | 720       | 527   | 630    |
| Vul (tgt)   | 1,601      | 966       | 670   | 767    |
| Non-Vul (h) | 714,790    | 466,179   | 352,675| 330,753|
| Non-Vul (t) | 476,528    | 201,391   | 139,578| 276,639|

**Metrics**: Standard precision, recall, and F1; AUPRC and AUROC (addressing extreme class imbalance); Macro-F1 for multi-class CWE-ID prediction.

**Baselines**: Logistic Regression, MLP, MemVul; zero-shot LLMs (LLaMA-2-13B, GPT-3, ChatGPT) with and without reasoning extensions (CoT-SC, ReAct, DS-Agent).

| Model Category   | Method     | Time(s/IR) | Precision | Recall | F1    | AUROC | AUPRC | Macro-F1 |
|------------------|------------|------------|-----------|--------|-------|-------|-------|----------|
| LLaMA-2          | +VulRTex   | 7.9        | 74.5      | 75.3   | 74.9  | 97.9  | 65.9  | 70.4     |
| ChatGPT          | +VulRTex   | 11.7       | 90.2      | 87.7   | 88.9  | 99.2  | 80.4  | 83.7     |

VulRTex-augmented ChatGPT achieved F1 = 88.9%, AUPRC = 80.4%, and Macro-F1 = 83.7%, outperforming the best baseline by +11.0%, +20.2%, and +10.5% respectively. Reasoning latency was approximately half that of ReAct or DS-Agent.

Ablations revealed critical performance drops without [SCR] analysis (−17.6% F1), random-walk pruning (−19.2% F1), or fact-correction (−12.3% F1).

## 6. Real-World Deployment and Observed Effectiveness

Tracking 2,098 new IRs in 2024 across ten major OSS projects (e.g., React, Kubernetes, Hugo), VulRTex identified 30 vulnerability-related IRs; 11 (36.7%) of these were assigned CVE-IDs prior to public disclosure (mean lead time, 35 days). In several cases (e.g., Carla #7025 → CWE-119 → CVE-2024-33903), VulRTex surfaced vulnerabilities over 100 days before CVE assignment, illustrating its operational viability.

| Project      | #2024 IRs | #Identified | #CVE-Assigned (%) |
|--------------|-----------|-------------|-------------------|
| Rimedo-ts    | 1         | 1           | 1/1 (100%)        |
| Carla        | 300       | 6           | 1/6 (16.7%)       |
| React        | 197       | 4           | 0/4 (0%)          |
| Kubernetes   | 597       | 3           | 1/3 (33.3%)       |
| **Total**    | 2,098     | 30          | 11/30 (36.7%)     |

A plausible implication is that integrating VulRTex into large-scale issue triage pipelines could materially accelerate vulnerability disclosure and mitigation timelines across diverse OSS ecosystems.

## 7. Limitations and Prospective Enhancements

Current limitations of VulRTex include the inability to process arbitrary video or audio streams, and dependence on the continued availability of IR page content (private/deleted IRs are out-of-scope). Directions for future improvement include integration with commit/patch analysis as additional reasoning evidence, extension to alternative collaborative platforms (e.g., GitLab, Gitee), and exploration of multimodal (e.g., CLIP-based) embeddings for richer screenshot processing.

VulRTex is, to date, the first framework to combine chain-of-thought LLM reasoning with retrieval-augmented, case-driven vulnerability graph retrieval for the practical and precise identification of vulnerabilities in rich-text OSS issue reports [2509.03875].

Source: https://www.emergentmind.com/topics/vulrtex