ProReFiCIA: LLM for Impact Analysis
- ProReFiCIA is an LLM-driven approach that automatically identifies impacted requirements using a zero-shot prompting, refinement, and NLI filtering workflow.
- It achieves high recall (up to 95.8%) while drastically reducing the analyst’s review set, with costs as low as 2.1% on industry datasets.
- The method combines precise prompt design with a human-in-loop post-processing pipeline to address semantic relations and cost-effective requirements change impact analysis.
Searching arXiv for the ProReFiCIA paper and closely related requirements change impact analysis work. ProReFiCIA is an LLM-driven approach for automatically identifying the impacted requirements when changes occur in requirements change impact analysis (CIA) (Etezadi et al., 31 Oct 2025). Its name expands to Prompt-Refinement-Filtering for CIA, reflecting a workflow that combines zero-shot prompting, a refinement pass over initially missed requirements, and filtering based on ranking and a fine-tuned NLI model. The approach is motivated by the fact that requirements are expressed in natural language, that impacted requirements may be related semantically rather than by exact wording, and that manual CIA is both effort-intensive and error-prone. In the reported evaluation, ProReFiCIA achieves a recall of 93.3% on the WASP benchmark and 95.8% on the SAT-DLink industry dataset, while reducing the analyst’s review set to 8.6% and 2.1% of all requirements, respectively (Etezadi et al., 31 Oct 2025).
1. Problem setting and rationale
Requirements change impact analysis asks which requirements are likely impacted by a given change rationale and therefore need review and update. The problem is difficult because impacted items may depend on semantic relationships and indirect impacts, rather than on lexical overlap alone. The paper frames manual CIA as costly, since overlooked impacts can propagate into downstream design, implementation, verification, and compliance issues (Etezadi et al., 31 Oct 2025).
A central observation is that large requirements collections make exhaustive inspection impractical. In the industry dataset used in the study, the system contains 192 requirements, yet only 18% are impacted by any given change rationale (Etezadi et al., 31 Oct 2025). This makes CIA a high-recall screening problem: the analyst must avoid missed impacts while minimizing the number of requirements that require manual review.
Within this setting, ProReFiCIA treats LLMs as a mechanism for reasoning over natural-language rationales and requirements more effectively than classical lexical matching or retrieval-only methods. The reported design goal is not fully autonomous decision-making, but generation of a smaller candidate set for human review (Etezadi et al., 31 Oct 2025). This suggests a human-in-the-loop operating model in which recall is prioritized and review effort is explicitly measured.
2. Architecture and workflow
ProReFiCIA consists of two main stages: Prompting with an LLM over the full requirements list and post-processing, where post-processing is divided into refinement and filtering (Etezadi et al., 31 Oct 2025). The paper emphasizes that the approach is zero-shot, does not require a pre-existing training dataset for the LLM component, and is intended to be portable across domains (Etezadi et al., 31 Oct 2025).
In the first stage, the LLM receives the entire requirements list at once in a cache-augmented generation (CAG) style setup. The initial output is a list of candidate impacted requirements, each accompanied by a justification. This design avoids many separate queries and gives the model simultaneous access to all candidates, in contrast to RAG-style retrieval pipelines that may exclude relevant requirements before reasoning begins (Etezadi et al., 31 Oct 2025).
The second stage begins with refinement. The authors report that LLMs can miss impacted requirements, especially when the list is long or the relevant item is not salient. To address this, ProReFiCIA takes the requirements not selected in the first pass, reruns the same prompt template, and merges newly identified requirements with the original impact set. The paper explicitly links this step to mitigation of the lost-in-the-middle problem (Etezadi et al., 31 Oct 2025).
The final step is filtering. After refinement, the candidate set may contain false positives. ProReFiCIA therefore asks the LLM to rank the predicted impacted requirements by confidence or strength of relation to the change rationale, considering the rationale, each requirement text, and the justification produced earlier. The ranked list is then processed by a fine-tuned NLI model that classifies whether the requirement-plus-justification entails the change rationale. The NLI output is binary: 1 = impacted / entailment and 0 = not impacted / non-entailment, with the third standard NLI label discarded because the task is binary (Etezadi et al., 31 Oct 2025).
The filtering rule is deliberately recall-preserving. If the impact set contains 5 or fewer items, all items are retained. Otherwise, a candidate is kept if the NLI predicts 1; if the NLI predicts 0, it is still kept when it appears in the top half of the ranked list; remaining items are removed (Etezadi et al., 31 Oct 2025). A plausible implication is that the filter is designed less as a strict verifier than as a conservative false-positive reducer.
3. Prompt design and model configuration
The prompting component is built through systematic prompt engineering rather than ad hoc template selection. The authors generate 64 prompt variants using the RICE template: Role, Instruction, Context, and Examples. Because the method uses zero-shot prompting only, the Examples component is excluded (Etezadi et al., 31 Oct 2025).
The paper enumerates seven prompt elements, combining mandatory and optional details:
- $\circled{1}$: Role — ask the model to act as a requirements engineer
- $\colorcircled{2}$: mandatory core instruction — identify impacted requirements
- $\circled{3}$: instruction to use commonsense knowledge
- $\circled{4}$: instruction to identify semantic relationships between keywords in rationale and requirements
- $\circled{5}$: context describing the CIA task
- $\circled{6}$: context describing what a change rationale is
- $\circled{7}$: context specifying the domain (Etezadi et al., 31 Oct 2025)
The 64 variants arise from all combinations of the optional prompt details around the mandatory instruction. Prompt sensitivity is therefore treated as a first-class empirical variable rather than an implementation detail (Etezadi et al., 31 Oct 2025).
Five LLMs are evaluated: GPT-4o, DeepSeek-R1, LLaMA-3-405B, Mistral Large, and Gemini-2.0. Reported implementation settings are Python 3.10.12, temperature = 0, seed = 16, frequency penalty = 0, and presence penalty = 0 (Etezadi et al., 31 Oct 2025). The study states that larger or stronger versions were chosen where possible.
The selected best general-purpose combination is LLaMA + P30, where P30 = $\circled{1}\colorcircled{2}\circled{5}\circled{6}$. This prompt includes role, the mandatory CIA instruction, CIA task context, and a change rationale definition, while excluding prompt details that were found to be harmful (Etezadi et al., 31 Oct 2025). The paper states that P30 is consistently in the top 10%, generalizes well across both datasets, and is especially strong on the more difficult SAT-DLink dataset (Etezadi et al., 31 Oct 2025).
4. Formalization, metrics, and empirical setting
The study defines effectiveness as average recall over change rationales:
where is the number of change rationales, $\colorcircled{2}$0 the true positives for rationale $\colorcircled{2}$1, and $\colorcircled{2}$2 the false negatives for rationale $\colorcircled{2}$3 (Etezadi et al., 31 Oct 2025).
It defines cost as the average fraction of the full requirements set that the analyst must inspect:
$\colorcircled{2}$4
where $\colorcircled{2}$5 is the total number of requirements and $\colorcircled{2}$6 the false positives for rationale $\colorcircled{2}$7 (Etezadi et al., 31 Oct 2025). This operationalization is central to the method’s design: ProReFiCIA is evaluated not only by recall and precision, but by how small a review set it produces.
For prompt-detail sensitivity analysis, the paper uses a gradient boosting model and defines the contribution of a detail $\colorcircled{2}$8 for a single tree as:
$\colorcircled{2}$9
with $\circled{3}$0 the number of samples at node $\circled{3}$1, $\circled{3}$2 the MSE at node $\circled{3}$3, and $\circled{3}$4 the left and right children (Etezadi et al., 31 Oct 2025). In the paper’s interpretation, a prompt detail is important if splitting on it substantially reduces prediction error.
Two datasets are used:
| Dataset | Requirements | Change rationales |
|---|---|---|
| WASP | 72 | 5 |
| SAT-DLink | 192 | 11 |
The WASP dataset is a public benchmark on mobile service platform requirements, with 22 impacted requirements total and an impacted proportion of 31%. SAT-DLink is a newly created industry dataset on a satellite data-link management system, with 35 impacted requirements total and an impacted proportion of 18% (Etezadi et al., 31 Oct 2025). The paper treats SAT-DLink as harder because of its larger size, smaller impacted fraction, and longer context.
The evaluation protocol is organized by research questions. RQ1 runs 64 prompts Ă— 5 models Ă— 2 datasets and analyzes box plots of F2. RQ2 focuses on GPT-4o and LLaMA for prompt-detail importance. RQ3 selects a model-prompt pair that generalizes across datasets. RQ4 studies refinement and filtering on the selected pair. RQ5 compares ProReFiCIA against baselines using recall/effectiveness and cost (Etezadi et al., 31 Oct 2025).
5. Main results and comparative performance
Across the full prompt sweep, the paper concludes that GPT-4o and LLaMA are the most robust, and that LLaMA is more stable than GPT-4o (Etezadi et al., 31 Oct 2025). Average results across all prompts on WASP are reported as follows: GPT-4o recall 82.7, precision 65.0, F2 77.1; DeepSeek recall 90.6, precision 31.4, F2 62.9; LLaMA recall 85.7, precision 51.2, F2 74.9; Mistral recall 87.0, precision 33.7, F2 65.2; Gemini recall 82.0, precision 58.8, F2 73.7 (Etezadi et al., 31 Oct 2025). On SAT-DLink, the averages are GPT-4o recall 64.5, precision 67.4, F2 62.9; DeepSeek recall 80.5, precision 28.9, F2 57.3; LLaMA recall 73.3, precision 43.1, F2 63.6; Mistral recall 75.7, precision 10.7, F2 31.7; Gemini recall 72.7, precision 31.2, F2 53.0 (Etezadi et al., 31 Oct 2025).
Prompt-detail analysis finds that Detail 3 and Detail 7 are consistently influential and generally hurt performance, with Detail 3 having the strongest negative effect (Etezadi et al., 31 Oct 2025). For GPT-4o, details 5 and 6 are most beneficial; for LLaMA, details 3 and 4 are the most influential, but negatively so (Etezadi et al., 31 Oct 2025). This leads to the pruning decision that supports the final P30 prompt.
The paper’s principal performance results are obtained with LLaMA-P30 and post-processing. On WASP, without post-processing the method reports TP = 19, FN = 3, FP = 13, effectiveness = 93.3%, and cost = 9.0%. With refinement, the values are TP = 19, FN = 3, FP = 30, effectiveness = 93.3%, and cost = 13.6%. With refinement + filtering, the values become TP = 19, FN = 3, FP = 12, effectiveness = 93.3%, and cost = 8.6% (Etezadi et al., 31 Oct 2025). Thus, on WASP, post-processing preserves recall and slightly reduces review cost.
On SAT-DLink, without post-processing the system yields TP = 26, FN = 7, FP = 12, effectiveness = 83.3%, and cost = 1.8%. With refinement, the results improve to TP = 32, FN = 1, FP = 51, effectiveness = 96.7%, and cost = 3.9%. With refinement + filtering, the reported values are TP = 30, FN = 3, FP = 17, effectiveness = 95.8%, and cost = 2.1% (Etezadi et al., 31 Oct 2025). The paper identifies this as the main result: SAT-DLink effectiveness improves by about 12.5 percentage points, from 83.3% to 95.8%, while cost remains very low at 2.1% (Etezadi et al., 31 Oct 2025).
The baseline comparison situates ProReFiCIA against NARCIA, CoT + RAG, and semantic-similarity variants S_T1, S_T2, and S_T3. On WASP, NARCIA obtains 97.8% effectiveness but at 22.2% cost, while ProReFiCIA attains 93.3% effectiveness at 8.6% cost (Etezadi et al., 31 Oct 2025). On SAT-DLink, CoT achieves 35.1% effectiveness at 8.1% cost; S_T1 reaches 86.3% effectiveness but at 99.0% cost; and ProReFiCIA reaches 95.8% effectiveness at 2.1% cost (Etezadi et al., 31 Oct 2025). The paper therefore characterizes ProReFiCIA as having a strong practical balance of high effectiveness and low review burden.
6. Interpretation, limitations, and significance
The reported findings support several substantive claims about the operating characteristics of ProReFiCIA. First, context-window prompting over the full requirements set is presented as preferable to iterative querying or RAG for this task, because retrieval can lose relevant items and iterative querying can create many false positives (Etezadi et al., 31 Oct 2025). Second, the study shows that prompt design matters: not all instructions that appear intuitively helpful improve performance, and some, such as explicit commonsense reasoning or domain context, can degrade it (Etezadi et al., 31 Oct 2025).
Third, the results indicate that refinement is especially valuable on more complex datasets. On SAT-DLink, refinement recovers missed impacted requirements that the first pass overlooks, albeit with a substantial rise in false positives, which the subsequent filtering stage partially corrects (Etezadi et al., 31 Oct 2025). This suggests that the architecture is deliberately staged around a recall-first principle.
The paper also reports systematic false negatives. In WASP, one change rationale concerning removal of the service profile led to missed requirements that used service in a broader sense; the model failed to recognize the more specific meaning of service profile (Etezadi et al., 31 Oct 2025). In SAT-DLink, some impacted requirements tied to launch-phase changes and SNMP-to-SDN monitoring changes were missed even after refinement and filtering, and some were missed across all 64 prompt variants (Etezadi et al., 31 Oct 2025). These findings indicate limitations in semantic disambiguation and domain-specific scope reasoning.
Threats to validity are also explicitly noted. Internal validity is supported by controlled temperature and seed, systematic prompt construction, and evaluation across multiple models and prompts. External validity remains limited by the use of only two datasets, although they differ in domain and complexity. Reproducibility is supported through reported implementation settings and archived experiments and artifacts (Etezadi et al., 31 Oct 2025).
In requirements-engineering terms, ProReFiCIA is best understood as a high-recall, low-cost LLM-based CIA pipeline rather than a general traceability system or a fully autonomous impact analyzer (Etezadi et al., 31 Oct 2025). Its core contribution lies in the combination of carefully selected zero-shot prompts, a refinement pass to recover misses, ranking to prioritize candidates, and NLI-based filtering to reduce false positives (Etezadi et al., 31 Oct 2025). A plausible implication is that the method’s significance depends less on any single model and more on the orchestration of prompting and post-processing around a review-cost objective.