LRCTI: LLM-Based CTI Credibility Verification

• LRCTI is a large language model-based framework that restructures CTI verification into an iterative process combining summarization, adaptive evidence retrieval, and prompt-based NLI reasoning.
• It condenses lengthy CTI reports into concise, verifiable claims that guide targeted evidence retrieval and underpin transparent credibility assessments.
• Empirical results on CTI-200 and PolitiFact highlight LRCTI’s improved performance over static classification approaches, demonstrating enhanced precision and interpretability.

Searching arXiv for LRCTI and closely related CTI credibility verification work. Searching arXiv for "LRCTI cyber threat intelligence credibility verification". LRCTI is a LLM-based framework for multi-step evidence retrieval and reasoning in Cyber Threat Intelligence credibility verification. It is designed to determine whether a Cyber Threat Intelligence report—or, more precisely, the key threat claim distilled from that report—is credible or incredible, while also producing an interpretable explanation grounded in retrieved evidence. The framework was introduced in “LRCTI: A LLM-Based Framework for Multi-Step Evidence Retrieval and Reasoning in Cyber Threat Intelligence Credibility Verification” (Tang et al., 15 Jul 2025). Its defining characteristic is that it does not treat credibility verification as a static classification problem; instead, it organizes the task as an iterative pipeline of claim-focused summarization, adaptive evidence retrieval, and prompt-based Natural Language Inference reasoning.

1. Conceptual scope and problem formulation

LRCTI addresses Cyber Threat Intelligence credibility verification, a task the paper treats as essential for reliable cybersecurity defense. The motivation is that traditional approaches generally formulate the task as a static classification problem, relying on handcrafted features, source heuristics, or supervised neural models that directly label a report or claim. The framework is introduced against three weaknesses attributed to those earlier approaches: brittleness under incomplete, heterogeneous, or noisy CTI; limited interpretability; and weak support for iterative evidence gathering (Tang et al., 15 Jul 2025).

The framework takes as input a raw CTI report and produces a binary credibility decision together with an explanation. In the notation of the paper, the output label is

$\hat{y} \in \{\text{credible}, \text{incredible}\}$

and the explanation is denoted $R_e$, with

$R_e = L(C, E_s).$

Here $C$ is the concise claim distilled from the report, $E_s$ is the selected evidence set, and $L$ is the LLM-based reasoning function (Tang et al., 15 Jul 2025).

A central feature of LRCTI is that the system operates through an intermediate three-way reasoning state,

$\hat{z} \in \{\text{Credible, Incredible, NEI}\},$

where NEI denotes “Not Enough Information.” This intermediate label is not the final benchmark output; rather, it functions as a control signal that determines whether the system should trigger another retrieval cycle. This suggests that LRCTI is best understood as an iterative retrieve–reason–refine framework rather than a one-pass retrieval-augmented classifier (Tang et al., 15 Jul 2025).

2. End-to-end architecture

The LRCTI pipeline consists of three modules executed in sequence: text summarization or claim extraction, adaptive multi-step evidence retrieval, and prompt-based Natural Language Inference reasoning. The paper describes the workflow in terms of a raw report first being reduced to concise verifiable claims $\mathbf{C}$, then mapped to a set of relevant paragraphs

$\mathbf{P = \{P_1, P_2, P_3, \ldots, P_k\}},$

from which sentence-level evidence is selected as

$$\mathbf{E}_\mathbf{s} = \{\mathbf{E}_1, \mathbf{E}_2, \mathbf{E}_3, \ldots, \mathbf{E}_j\}.$$

A LLM then reasons over the claim and the evidence to generate the credibility label and explanation (Tang et al., 15 Jul 2025).

The interaction between these modules is explicitly iterative. The summarization stage compresses a long CTI report into a compact claim suitable for downstream retrieval. The retrieval module searches a CTI-specific corpus for relevant paragraphs and sentences. The reasoning module evaluates whether the evidence supports, refutes, or is insufficient for the claim. If the evidence is judged irrelevant, insufficient, or low-confidence, the framework does not terminate. Instead, it performs re-retrieval guided by earlier reasoning outputs, established evidence, and updated claims (Tang et al., 15 Jul 2025).

The procedure described in the paper can be summarized as follows. A CTI report is provided as input; salient report sentences are scored and selected; the LLM summarizes them into a concise verifiable claim $R_e$0; the system retrieves top-$R_e$1 paragraphs; it progressively refines paragraph candidates; it selects top-$R_e$2 evidence sentences; then the LLM reasons over $R_e$3, producing a provisional label $R_e$4, an explanation $R_e$5, and a confidence score $R_e$6. If $R_e$7 or the confidence is too low, the framework stores validated evidence, generates updated claims, and repeats retrieval (Tang et al., 15 Jul 2025).

The paper indicates that a retrieval-confidence threshold of 50% is used operationally. It also states that an overconfidence correction factor $R_e$8 is applied to raw LLM confidence, and that re-retrieval is triggered when $R_e$9. The text also gives the generic formulation

$R_e = L(C, E_s).$0

with $R_e = L(C, E_s).$1. The coexistence of the generic threshold notation and the explicit 50% trigger is described in the paper itself and constitutes a small internal inconsistency rather than an external reinterpretation (Tang et al., 15 Jul 2025).

3. Summarization and claim extraction

A major design choice in LRCTI is that it does not submit the full CTI report directly to retrieval and reasoning. Instead, it first derives a concise, verifiable claim. The paper argues that raw CTI reports are frequently long, technical, redundant, and noisy, and that whole-report processing can lower retrieval precision by including content irrelevant to the verifiable core proposition (Tang et al., 15 Jul 2025).

Let the report sentences be

$R_e = L(C, E_s).$2

Each sentence is assigned an importance score

$R_e = L(C, E_s).$3

where $R_e = L(C, E_s).$4 measures overlap with the whole document, $R_e = L(C, E_s).$5 is a positional weight, $R_e = L(C, E_s).$6 is semantic similarity with other sentences, and $R_e = L(C, E_s).$7 are hyperparameters (Tang et al., 15 Jul 2025). The top $R_e = L(C, E_s).$8 sentences form a subset $R_e = L(C, E_s).$9, which is converted into an LLM prompt $C$0, yielding the concise claim

$C$1

The framework uses a prompt-based summarization approach rather than fine-tuning a summarization model such as Pegasus. The appendix reportedly uses a “Text summary (ts) prompt” to convert long report content into verifiable short claims, although the exact prompt text is not reproduced in the paper body (Tang et al., 15 Jul 2025). A plausible implication is that reproducibility of the summarization stage is limited by the absence of exact prompt text and explicit hyperparameter values for $C$2, $C$3, and $C$4.

The notation is slightly inconsistent about whether LRCTI extracts one claim or several. The paper initially refers to concise claims as $C$5, but most of the method description uses a singular claim $C$6. The most defensible reading is that the framework initially operates on one compact claim per report, while later iterations may generate updated or simplified sub-claims to guide re-retrieval (Tang et al., 15 Jul 2025).

4. Adaptive multi-step evidence retrieval

The retrieval component is the framework’s central innovation. Rather than performing a single retrieval pass, LRCTI uses an adaptive multi-step mechanism in which LLM feedback about evidence sufficiency and confidence determines whether retrieval should continue (Tang et al., 15 Jul 2025).

The retrieval corpus is CTI-domain-specific and is constructed through web scraping, data cleaning and formatting, and manual verification and annotation. Sources include ATT{paper_content}CK framework, Wikipedia, and prominent security blogs. The paper states that dynamic content is handled through JavaScript rendering and session management, and that the resulting corpus is normalized into a structured format with titles, links, and body content. Duplicate reports are removed, and related reports about the same event are merged (Tang et al., 15 Jul 2025).

Paragraph retrieval uses a dense retriever. Given a claim $C$7 and paragraph corpus $C$8, the retrieved paragraph set is

$C$9

where $E_s$0 denotes claim–paragraph similarity (Tang et al., 15 Jul 2025). The exact retriever architecture is not named in the paper. The experimental details specify a 20-sentence window and 50 candidate paragraphs per claim.

After initial dense retrieval, LRCTI applies progressive paragraph selection. A sliding window over the current paragraph set generates new candidates $E_s$1. From the union $E_s$2, the LLM selects the top-$E_s$3 most relevant paragraphs with respect to the claim, and the paragraph set is updated while kept at fixed size (Tang et al., 15 Jul 2025). The paper presents this as more than simple reranking, because the working paragraph set is revised to reduce redundancy and improve contextual coverage.

Sentence-level evidence extraction follows paragraph refinement. For each sentence $E_s$4 in the refined paragraph set, a relevance score $E_s$5 is computed relative to the claim, and the top-$E_s$6 sentences form the evidence set

$E_s$7

Again, the exact encoder and similarity function are not provided in the paper (Tang et al., 15 Jul 2025).

The retrieval loop is controlled by the reasoning module, which outputs

$E_s$8

Re-retrieval is triggered when evidence is irrelevant, insufficient, or low-confidence. Formally,

$E_s$9

continues the loop in the generic description, while the implementation states that if $L$0, the next iteration is triggered automatically (Tang et al., 15 Jul 2025).

An important memory mechanism is the use of “established evidence” and “updated claims.” Established evidence is previously validated, compressed evidence retained across iterations; updated claims are simplified or reformulated claims used to guide the next retrieval stage. This suggests that LRCTI treats retrieval as stateful search rather than repeated querying with the original claim only (Tang et al., 15 Jul 2025).

The paper’s sensitivity analysis shows that performance improves as the number of retrieval steps increases, but plateaus after three retrieval steps and may slightly decline beyond that. The paper therefore concludes that three retrieval steps are optimal across both datasets (Tang et al., 15 Jul 2025).

5. Prompt-based NLI reasoning and interpretability

The final stage of LRCTI frames credibility verification as an NLI-style problem in which evidence either supports, refutes, or is insufficient for a claim. The internal label space is

$L$1

while the benchmark output remains binary,

$L$2

Thus NEI acts as an intermediate reasoning state rather than a final user-facing label (Tang et al., 15 Jul 2025).

The reasoning prompt aggregates the selected evidence set and the claim, and asks the LLM to determine whether the evidence is sufficient and what credibility label follows. The appendix reportedly uses a “Step check (sc) prompt” and a “Response correction (rc) prompt.” The response-correction prompt is intended to reduce self-contradictory answers, while the step-check prompt is part of the iterative reasoning process (Tang et al., 15 Jul 2025).

At each reasoning stage, the LLM produces a label, a rationale or explanation, and a confidence score. If the label is credible or incredible and the confidence is high enough, the system terminates with that decision. Otherwise, the retrieval module is re-invoked. This means the justification is not merely a post-hoc explanation layer; it is generated as part of the decision process itself (Tang et al., 15 Jul 2025).

Interpretability is one of LRCTI’s principal claimed advantages. The paper states that the framework “clearly distinguish[es] between retrieved factual information and LLM-generated inferences,” and presents this as an attribution mechanism supporting transparency and user trust (Tang et al., 15 Jul 2025). A plausible implication is that LRCTI’s explainability is tied less to formal rationale metrics and more to source-grounded evidence presentation plus explicit reasoning traces.

The framework’s interpretability claims are further supported by a user study. Forty claims from CTI-200 were used, with 20 credible and 20 incredible claims, and 8 university students judged randomized claim–evidence pairs within 3 minutes while also rating confidence on a 5-point Likert scale. LRCTI evidence was compared with MUSER evidence. The reported results are:

Method	F1	Precision	Agreement
MUSER	0.758	0.733	76.7%
LRCTI	0.815	0.812	81.6%

Here Agreement is the proportion of concurrence between users’ judgments and the model’s judgment (Tang et al., 15 Jul 2025). The paper does not report explanation faithfulness or sufficiency metrics.

6. Experimental evaluation and reported performance

LRCTI is evaluated on two benchmark datasets: CTI-200 and PolitiFact. The paper describes CTI-200 as a cyber threat intelligence dataset collected from multiple cybersecurity websites and labeled credible or incredible. However, the table in the paper reports 547 credible and 453 incredible examples, for a total of 1000, despite the dataset name “CTI-200” (Tang et al., 15 Jul 2025). This discrepancy appears in the paper itself.

PolitiFact is taken from FakeNewsNet and contains political news articles labeled real or fake based on professional fact-checker judgments. The reported statistics are 399 real and 345 fake articles, totaling 744 (Tang et al., 15 Jul 2025).

The benchmark task is treated as binary classification. Baselines include evidence-based methods—DeClarE, HAN, EHIAN, MAC, GET, and MUSER—and LLM-based methods—GPT-3.5-turbo, ChatGLM2-6B, WEBGLM-2B, and ProgramFC (Tang et al., 15 Jul 2025). LRCTI uses OpenAI GPT-3.5-turbo as backbone, with temperature 0, top-p 0.75, token limit 4096, overconfidence correction factor $L$3, paragraph retrieval with a 20-sentence window, and 50 candidate paragraphs per claim (Tang et al., 15 Jul 2025).

On CTI-200, the reported results are:

Method	F1-Ma	F1-Mi
DeClarE	0.725	0.786
HAN	0.752	0.802
EHIAN	0.784	0.828
MAC	0.786	0.833
GET	0.800	0.846
MUSER	0.858	0.894
GPT-3.5-turbo	0.782	0.798
ChatGLM2-6B	0.780	0.795
WEBGLM-2B	0.840	0.855
ProgramFC	0.850	0.865
LRCTI	0.909	0.936

The more detailed CTI-200 class-wise figures for LRCTI are F1-T 0.869, P-T 0.909, R-T 0.833, F1-F 0.949, P-F 0.933, and R-F 0.965 (Tang et al., 15 Jul 2025).

On PolitiFact, the reported results are:

Method	F1-Ma	F1-Mi
DeClarE	0.654	0.651
HAN	0.661	0.660
EHIAN	0.664	0.663
MAC	0.678	0.675
GET	0.694	0.692
MUSER	0.732	0.729
GPT-3.5-turbo	0.567	0.553
ChatGLM2-6B	0.522	0.515
WEBGLM-2B	0.628	0.633
ProgramFC	0.684	0.678
LRCTI	0.861	0.861

The more detailed PolitiFact class-wise figures for LRCTI are F1-T 0.883, P-T 0.899, R-T 0.870, F1-F 0.838, P-F 0.842, and R-F 0.835 (Tang et al., 15 Jul 2025).

The abstract summarizes the CTI-200 result as improving F1-Macro and F1-Micro by over 5%, reaching 90.9% and 93.6%, respectively (Tang et al., 15 Jul 2025). Compared with MUSER on CTI-200, LRCTI improves F1-Macro by 0.051 absolute and F1-Micro by 0.042 absolute. On PolitiFact, the gains over MUSER are 0.129 and 0.132 absolute, respectively (Tang et al., 15 Jul 2025).

The paper also includes an ablation summary. Removing re-retrieval significantly hurts performance, and removing summarization also causes a noticeable drop; the full LRCTI performs best. Although the exact ablation values are not printed in the text, the paper interprets the results as showing that summarization improves signal-to-noise ratio, while iterative retrieval improves evidence completeness and precision (Tang et al., 15 Jul 2025).

A corpus comparison is also reported. Using Wikipedia versus the CTI-specific corpus yields:

Corpus	Efficiency	$L$4
Wiki	10s	0.733
CTI	5s	0.895

This indicates that the CTI corpus is both faster and more semantically aligned for CTI verification (Tang et al., 15 Jul 2025).

7. Significance, limitations, and relation to adjacent work

The paper attributes LRCTI’s performance gains to three interacting factors. First, claim-focused summarization reduces irrelevant report content before retrieval. Second, iterative retrieval improves evidence completeness and precision when one-shot retrieval is insufficient. Third, NLI-style reasoning grounds the decision in evidence and can explicitly represent insufficiency through NEI (Tang et al., 15 Jul 2025). This suggests that LRCTI’s contribution is architectural rather than merely model-scale-driven.

The framework also has clear limitations. It depends heavily on the quality of the CTI corpus; if that corpus is incomplete, outdated, noisy, or biased, retrieval quality will suffer. Retrieval errors can propagate into downstream reasoning. The authors explicitly note that LLMs can be inconsistent and overconfident; the response-correction prompt and the confidence damping factor $L$5 are mitigation measures, not guarantees. Multi-step retrieval and repeated LLM calls likely increase runtime and API cost, although the paper does not quantify total cost. Reproducibility is also limited by missing details on retriever architecture, indexing, hardware, train/validation/test splits, and exact prompts (Tang et al., 15 Jul 2025).

Within the broader literature represented in the data, LRCTI differs from static evidence-based systems such as DeClarE, HAN, EHIAN, MAC, GET, and MUSER by treating evidence acquisition as an adaptive process. It also differs from direct-prompt LLM baselines such as GPT-3.5-turbo and ChatGLM2-6B by explicitly structuring the workflow around claim extraction, retrieval refinement, and evidence-grounded reasoning rather than using the model as an end-to-end classifier (Tang et al., 15 Jul 2025). This suggests that the main novelty lies in procedural decomposition and stateful evidence accumulation, not solely in the use of an LLM.

A common misconception would be to interpret LRCTI as just another retrieval-augmented generation system. The paper’s description does not support that simplification. Retrieval is not one-shot, the query is not fixed, and the reasoning stage can explicitly defer judgment through NEI. Another misconception would be to regard the reported robustness to noisy and heterogeneous CTI as exhaustively established. The paper supports this claim indirectly through benchmark gains, retrieval sensitivity analysis, corpus comparison, and user-study agreement, but it does not run explicit corruption or perturbation stress tests (Tang et al., 15 Jul 2025).

In summary, LRCTI is an LLM-based credibility-verification framework for CTI that organizes verification into a sequence of summarization, adaptive evidence retrieval, and NLI-style reasoning. Its core methodological contribution is the replacement of one-shot classification with iterative evidence acquisition and claim refinement. Empirically, the framework achieves the best reported results in the paper on CTI-200 and PolitiFact, reaching 90.9% F1-Macro and 93.6% F1-Micro on CTI-200, while also improving user-facing evidence quality in a human study (Tang et al., 15 Jul 2025).