---
title: 'ProReFiCIA: LLM for Impact Analysis'
url: https://www.emergentmind.com/topics/proreficia
type: topic
---

# ProReFiCIA: LLM for Impact Analysis

Searching arXiv for the ProReFiCIA paper and closely related requirements change impact analysis work.
ProReFiCIA is an **LLM-driven approach for automatically identifying the impacted requirements when changes occur** in **requirements change impact analysis (CIA)** [2511.00262]. Its name expands to **Prompt-Refinement-Filtering for CIA**, reflecting a workflow that combines **zero-shot prompting**, a **refinement** pass over initially missed requirements, and **filtering** based on ranking and a **fine-tuned NLI model**. The approach is motivated by the fact that requirements are expressed in **natural language**, that impacted requirements may be related **semantically** rather than by exact wording, and that manual CIA is both **effort-intensive** and **error-prone**. In the reported evaluation, ProReFiCIA achieves a recall of **93.3%** on the **WASP** benchmark and **95.8%** on the **SAT-DLink** industry dataset, while reducing the analyst’s review set to **8.6%** and **2.1%** of all requirements, respectively [2511.00262].

## 1. Problem setting and rationale

Requirements change impact analysis asks which requirements are likely impacted by a given **change rationale** and therefore need review and update. The problem is difficult because impacted items may depend on **semantic relationships** and **indirect impacts**, rather than on lexical overlap alone. The paper frames manual CIA as **costly**, since overlooked impacts can propagate into downstream **design, implementation, verification, and compliance issues** [2511.00262].

A central observation is that large requirements collections make exhaustive inspection impractical. In the industry dataset used in the study, the system contains **192 requirements**, yet only **18%** are impacted by any given change rationale [2511.00262]. This makes CIA a high-recall screening problem: the analyst must avoid missed impacts while minimizing the number of requirements that require manual review.

Within this setting, ProReFiCIA treats LLMs as a mechanism for reasoning over natural-language rationales and requirements more effectively than classical lexical matching or retrieval-only methods. The reported design goal is not fully autonomous decision-making, but generation of a **smaller candidate set for human review** [2511.00262]. This suggests a human-in-the-loop operating model in which recall is prioritized and review effort is explicitly measured.

## 2. Architecture and workflow

ProReFiCIA consists of two main stages: **Prompting with an LLM over the full requirements list** and **post-processing**, where post-processing is divided into **refinement** and **filtering** [2511.00262]. The paper emphasizes that the approach is **zero-shot**, **does not require a pre-existing training dataset** for the LLM component, and is intended to be portable across domains [2511.00262].

In the first stage, the LLM receives the **entire requirements list** at once in a **cache-augmented generation (CAG)** style setup. The initial output is a list of candidate impacted requirements, each accompanied by a justification. This design avoids many separate queries and gives the model simultaneous access to all candidates, in contrast to **RAG**-style retrieval pipelines that may exclude relevant requirements before reasoning begins [2511.00262].

The second stage begins with **refinement**. The authors report that LLMs can miss impacted requirements, especially when the list is long or the relevant item is not salient. To address this, ProReFiCIA takes the requirements **not selected** in the first pass, reruns the **same prompt template**, and merges newly identified requirements with the original impact set. The paper explicitly links this step to mitigation of the **lost-in-the-middle** problem [2511.00262].

The final step is **filtering**. After refinement, the candidate set may contain false positives. ProReFiCIA therefore asks the LLM to rank the predicted impacted requirements by confidence or strength of relation to the change rationale, considering the rationale, each requirement text, and the justification produced earlier. The ranked list is then processed by a **fine-tuned NLI model** that classifies whether the requirement-plus-justification **entails** the change rationale. The NLI output is binary: **`1` = impacted / entailment** and **`0` = not impacted / non-entailment**, with the third standard NLI label discarded because the task is binary [2511.00262].

The filtering rule is deliberately recall-preserving. If the impact set contains **5 or fewer** items, all items are retained. Otherwise, a candidate is kept if the NLI predicts **`1`**; if the NLI predicts **`0`**, it is still kept when it appears in the **top half** of the ranked list; remaining items are removed [2511.00262]. A plausible implication is that the filter is designed less as a strict verifier than as a conservative false-positive reducer.

## 3. Prompt design and model configuration

The prompting component is built through systematic prompt engineering rather than ad hoc template selection. The authors generate **64 prompt variants** using the **RICE** template: **Role**, **Instruction**, **Context**, and **Examples**. Because the method uses **zero-shot prompting only**, the **Examples** component is excluded [2511.00262].

The paper enumerates seven prompt elements, combining mandatory and optional details:

- **\(\circled{1}\)**: Role — ask the model to act as a requirements engineer  
- **\(\colorcircled{2}\)**: mandatory core instruction — identify impacted requirements  
- **\(\circled{3}\)**: instruction to use commonsense knowledge  
- **\(\circled{4}\)**: instruction to identify semantic relationships between keywords in rationale and requirements  
- **\(\circled{5}\)**: context describing the CIA task  
- **\(\circled{6}\)**: context describing what a change rationale is  
- **\(\circled{7}\)**: context specifying the domain [2511.00262]

The **64 variants** arise from all combinations of the optional prompt details around the mandatory instruction. Prompt sensitivity is therefore treated as a first-class empirical variable rather than an implementation detail [2511.00262].

Five LLMs are evaluated: **GPT-4o**, **DeepSeek-R1**, **LLaMA-3-405B**, **Mistral Large**, and **Gemini-2.0**. Reported implementation settings are **Python 3.10.12**, **temperature = 0**, **seed = 16**, **frequency penalty = 0**, and **presence penalty = 0** [2511.00262]. The study states that larger or stronger versions were chosen where possible.

The selected best general-purpose combination is **LLaMA + P30**, where **P30 = \(\circled{1}\colorcircled{2}\circled{5}\circled{6}\)**. This prompt includes **role**, the **mandatory CIA instruction**, **CIA task context**, and a **change rationale definition**, while excluding prompt details that were found to be harmful [2511.00262]. The paper states that **P30 is consistently in the top 10%**, generalizes well across both datasets, and is especially strong on the more difficult **SAT-DLink** dataset [2511.00262].

## 4. Formalization, metrics, and empirical setting

The study defines **effectiveness** as average recall over change rationales:

\[
eff = \frac{1}{N_c} \sum_{i=1}^{N_c} \frac{TP_{c_i}}{TP_{c_i} + FN_{c_i}}
\]

where \(N_c\) is the number of change rationales, \(TP_{c_i}\) the true positives for rationale \(i\), and \(FN_{c_i}\) the false negatives for rationale \(i\) [2511.00262].

It defines **cost** as the average fraction of the full requirements set that the analyst must inspect:

\[
cost = \frac{1}{N_c} \sum_{i=1}^{N_c} \frac{TP_{c_i} + FP_{c_i}}{N_{req}}
\]

where \(N_{req}\) is the total number of requirements and \(FP_{c_i}\) the false positives for rationale \(i\) [2511.00262]. This operationalization is central to the method’s design: ProReFiCIA is evaluated not only by recall and precision, but by how small a review set it produces.

For prompt-detail sensitivity analysis, the paper uses a gradient boosting model and defines the contribution of a detail \(d\) for a single tree as:

\[
\text{Imp}(d) = \frac{1}{N_{\text{root}}} \sum_{j \,\text{splits on } d} \Big( N_j I_j - N_{j_L} I_{j_L} - N_{j_R} I_{j_R} \Big)
\]

with \(N_j\) the number of samples at node \(j\), \(I_j\) the MSE at node \(j\), and \(j_L, j_R\) the left and right children [2511.00262]. In the paper’s interpretation, a prompt detail is important if splitting on it substantially reduces prediction error.

Two datasets are used:

| Dataset | Requirements | Change rationales |
|---|---:|---:|
| WASP | 72 | 5 |
| SAT-DLink | 192 | 11 |

The **WASP** dataset is a public benchmark on **mobile service platform requirements**, with **22 impacted requirements total** and an **impacted proportion of 31%**. **SAT-DLink** is a newly created industry dataset on a **satellite data-link management system**, with **35 impacted requirements total** and an **impacted proportion of 18%** [2511.00262]. The paper treats SAT-DLink as harder because of its larger size, smaller impacted fraction, and longer context.

The evaluation protocol is organized by research questions. **RQ1** runs **64 prompts × 5 models × 2 datasets** and analyzes **box plots of F2**. **RQ2** focuses on **GPT-4o** and **LLaMA** for prompt-detail importance. **RQ3** selects a model-prompt pair that generalizes across datasets. **RQ4** studies refinement and filtering on the selected pair. **RQ5** compares ProReFiCIA against baselines using **recall/effectiveness** and **cost** [2511.00262].

## 5. Main results and comparative performance

Across the full prompt sweep, the paper concludes that **GPT-4o and LLaMA are the most robust**, and that **LLaMA is more stable than GPT-4o** [2511.00262]. Average results across all prompts on **WASP** are reported as follows: GPT-4o recall **82.7**, precision **65.0**, F2 **77.1**; DeepSeek recall **90.6**, precision **31.4**, F2 **62.9**; LLaMA recall **85.7**, precision **51.2**, F2 **74.9**; Mistral recall **87.0**, precision **33.7**, F2 **65.2**; Gemini recall **82.0**, precision **58.8**, F2 **73.7** [2511.00262]. On **SAT-DLink**, the averages are GPT-4o recall **64.5**, precision **67.4**, F2 **62.9**; DeepSeek recall **80.5**, precision **28.9**, F2 **57.3**; LLaMA recall **73.3**, precision **43.1**, F2 **63.6**; Mistral recall **75.7**, precision **10.7**, F2 **31.7**; Gemini recall **72.7**, precision **31.2**, F2 **53.0** [2511.00262].

Prompt-detail analysis finds that **Detail 3** and **Detail 7** are consistently influential and generally **hurt performance**, with **Detail 3** having the strongest negative effect [2511.00262]. For **GPT-4o**, details **5** and **6** are most beneficial; for **LLaMA**, details **3** and **4** are the most influential, but negatively so [2511.00262]. This leads to the pruning decision that supports the final **P30** prompt.

The paper’s principal performance results are obtained with **LLaMA-P30** and post-processing. On **WASP**, without post-processing the method reports **TP = 19**, **FN = 3**, **FP = 13**, **effectiveness = 93.3%**, and **cost = 9.0%**. With **refinement**, the values are **TP = 19**, **FN = 3**, **FP = 30**, **effectiveness = 93.3%**, and **cost = 13.6%**. With **refinement + filtering**, the values become **TP = 19**, **FN = 3**, **FP = 12**, **effectiveness = 93.3%**, and **cost = 8.6%** [2511.00262]. Thus, on WASP, post-processing preserves recall and slightly reduces review cost.

On **SAT-DLink**, without post-processing the system yields **TP = 26**, **FN = 7**, **FP = 12**, **effectiveness = 83.3%**, and **cost = 1.8%**. With **refinement**, the results improve to **TP = 32**, **FN = 1**, **FP = 51**, **effectiveness = 96.7%**, and **cost = 3.9%**. With **refinement + filtering**, the reported values are **TP = 30**, **FN = 3**, **FP = 17**, **effectiveness = 95.8%**, and **cost = 2.1%** [2511.00262]. The paper identifies this as the main result: **SAT-DLink effectiveness improves by about 12.5 percentage points**, from **83.3%** to **95.8%**, while cost remains very low at **2.1%** [2511.00262].

The baseline comparison situates ProReFiCIA against **NARCIA**, **CoT + RAG**, and semantic-similarity variants **S\_T1**, **S\_T2**, and **S\_T3**. On **WASP**, **NARCIA** obtains **97.8%** effectiveness but at **22.2%** cost, while **ProReFiCIA** attains **93.3%** effectiveness at **8.6%** cost [2511.00262]. On **SAT-DLink**, **CoT** achieves **35.1%** effectiveness at **8.1%** cost; **S\_T1** reaches **86.3%** effectiveness but at **99.0%** cost; and **ProReFiCIA** reaches **95.8%** effectiveness at **2.1%** cost [2511.00262]. The paper therefore characterizes ProReFiCIA as having a strong practical balance of high effectiveness and low review burden.

## 6. Interpretation, limitations, and significance

The reported findings support several substantive claims about the operating characteristics of ProReFiCIA. First, **context-window prompting** over the full requirements set is presented as preferable to iterative querying or RAG for this task, because retrieval can lose relevant items and iterative querying can create many false positives [2511.00262]. Second, the study shows that **prompt design matters**: not all instructions that appear intuitively helpful improve performance, and some, such as explicit **commonsense reasoning** or domain context, can degrade it [2511.00262].

Third, the results indicate that **refinement** is especially valuable on more complex datasets. On SAT-DLink, refinement recovers missed impacted requirements that the first pass overlooks, albeit with a substantial rise in false positives, which the subsequent **filtering** stage partially corrects [2511.00262]. This suggests that the architecture is deliberately staged around a recall-first principle.

The paper also reports systematic false negatives. In **WASP**, one change rationale concerning removal of the **service profile** led to missed requirements that used **service** in a broader sense; the model failed to recognize the more specific meaning of **service profile** [2511.00262]. In **SAT-DLink**, some impacted requirements tied to **launch-phase changes** and **SNMP-to-SDN monitoring changes** were missed even after refinement and filtering, and some were missed across **all 64 prompt variants** [2511.00262]. These findings indicate limitations in semantic disambiguation and domain-specific scope reasoning.

Threats to validity are also explicitly noted. Internal validity is supported by controlled **temperature** and **seed**, systematic prompt construction, and evaluation across multiple models and prompts. External validity remains limited by the use of **only two datasets**, although they differ in domain and complexity. Reproducibility is supported through reported implementation settings and archived experiments and artifacts [2511.00262].

In requirements-engineering terms, ProReFiCIA is best understood as a **high-recall, low-cost LLM-based CIA pipeline** rather than a general traceability system or a fully autonomous impact analyzer [2511.00262]. Its core contribution lies in the combination of **carefully selected zero-shot prompts**, a **refinement pass** to recover misses, **ranking** to prioritize candidates, and **NLI-based filtering** to reduce false positives [2511.00262]. A plausible implication is that the method’s significance depends less on any single model and more on the orchestration of prompting and post-processing around a review-cost objective.

Source: https://www.emergentmind.com/topics/proreficia