Affective-ROPTester: ROP Risk & Bias Framework

• The paper introduces Affective-ROPTester, an evaluation framework assessing LLMs' risk prediction and emotional bias using admission notes.
• It employs three prompting schemes—Instruction, Chain-of-Thought, and In-Context Learning—to analyze structured reasoning versus example-based elicitation.
• Results show that structured CoT prompts notably reduce risk overestimation, highlighting the impact of affective framing on clinical NLP performance.

Searching arXiv for the cited benchmark paper and closely related methodological context. Affective-ROPTester is an automated evaluation framework for analyzing both the predictive capability and the affective bias of LLMs in retinopathy of prematurity (ROP) risk stratification from admission records. It was introduced together with CROP, a Chinese benchmark dataset of 993 admission notes annotated with low-, medium-, and high-risk labels, and it operationalizes evaluation through three prompting schemes—Instruction-based, Chain-of-Thought (CoT), and In-Context Learning (ICL)—augmented by negative, neutral, and positive affective framings. In this formulation, the framework is not merely a benchmark for clinical classification; it is also a prompt-level probe of how emotional framing alters error patterns, especially the tendency to overestimate medium- and high-risk cases (Zhao et al., 8 Jul 2025).

1. Clinical problem and evaluative scope

ROP is a retinal vascular disease affecting preterm and low-birth-weight infants and is potentially blinding. The paper emphasizes that its pathogenesis is still not fully understood, but it identifies clinically recognized risk factors including birth weight, gestational age, multiple gestation, mode of delivery, duration of oxygen therapy, oxygen concentration, red blood cell transfusion, and maternal conditions such as gestational hypertension and diabetes. Affective-ROPTester is centered on a harder task than diagnosis: predicting future ROP risk from admission notes alone, before disease is directly diagnosed (Zhao et al., 8 Jul 2025).

The framework has two explicit goals. The first is capability analysis: determining whether contemporary LLMs can stratify ROP risk from text-only admission records, and whether prompting methods materially improve that capability. The second is affective bias analysis: determining whether emotional prompt framing changes predictive performance and systematically shifts outputs toward pessimistic or optimistic risk assignments. In the paper’s terminology, the key issue is not demographic fairness but predictive bias, especially risk overestimation in low- and medium-risk cases (Zhao et al., 8 Jul 2025).

Affective-ROPTester treats prompt design as both an elicitation mechanism and a bias variable. This is methodologically distinctive. The Instruction scheme tests intrinsic knowledge, whereas CoT and ICL inject external medical structure. Emotional framing is then superimposed on this prompting layer to examine whether the model’s predictive boundary moves under negative, neutral, or positive affective instructions. The framework is therefore simultaneously a clinical NLP benchmark and an affect-sensitive audit apparatus.

2. CROP dataset and label construction

CROP was constructed from 3,922 original records provided by Guangdong Women and Children Hospital. Ethics approval was obtained from Shenzhen Eye Hospital under NO. 2022KYPJ064, and the study adhered to the Declaration of Helsinki. Because only 331 of the original records were high-risk samples, the dataset was balanced by selecting 331 low-risk, 331 medium-risk, and 331 high-risk admission notes, yielding 993 notes in total (Zhao et al., 8 Jul 2025).

The label mapping is retrospective and outcome-based. Final diagnosis groups are converted into benchmark risk labels as follows: No ROP becomes low-risk, Mild ROP becomes medium-risk, and Severe ROP becomes high-risk. Mild ROP corresponds to type 2 ROP, while Severe ROP includes type 1 ROP, aggressive ROP, and stages 4 and 5 ROP. Each benchmark item is an initial hospital admission note for a preterm infant, so the task is to infer later risk category from early textual evidence rather than to classify confirmed retinal findings (Zhao et al., 8 Jul 2025).

The paper states that the test set contains 331 notes, partitioned into 112 low-risk, 111 medium-risk, and 108 high-risk samples. It also reports summary statistics that align with clinical expectations: lower gestational age and lower birth weight are associated with higher risk.

Risk	Number	Gestational Age	Birth Weight
Low	112	32.4 weeks	1.6 kg
Medium	111	29.3 weeks	1.3 kg
High	108	28.5 weeks	1.1 kg

The notes contain clinically relevant textual content from which variables such as gestational age, birth weight, multiple pregnancies, oxygen exposure, pregnancy complications, and general neonatal history can be inferred or read. The paper repeatedly identifies gestational age and birth weight as the most important and reliably useful signals. At the same time, it does not provide a detailed preprocessing pipeline, de-identification procedure, or inter-annotator agreement protocol, which limits exact reconstruction of corpus preparation (Zhao et al., 8 Jul 2025).

3. Prompting architecture and affective manipulation

Affective-ROPTester compares three prompting schemes. The Instruction scheme is the baseline and is designed to test the model’s intrinsic medical knowledge without retrieval or explicit reasoning scaffolds. The representative role instruction is: “You are an ophthalmology specialist for premature infants. Please classify whether the following premature infants are likely to develop retinopathy of prematurity.” Its input form is given as

$x' = [S, U(x)],$

where $S$ is the predefined system instruction and $U(x)$ is the user input containing the note and classification request (Zhao et al., 8 Jul 2025).

The CoT scheme extends this baseline with a predefined eight-step reasoning template: identify basic data and risk factors; analyze gestational age and birth weight; consider multiple pregnancies; assess oxygen exposure; analyze pregnancy complications; consider growth factors; perform comprehensive assessment; and make final risk classification. Its input form is

$x' = [S, \text{CoT} + U(x)].$

This is not free-form reasoning but a medically curated checklist intended to align the model’s inference path with known ROP risk factors (Zhao et al., 8 Jul 2025).

The ICL scheme supplies retrieved demonstrations from the CROP training pool. Retrieval is based on BM25, with the score written as

$$score(x,d) = \begin{matrix}\sum_i^n W_iR(x_i,d)\end{matrix},$$

where $x_i$ is a word in sample $x$, $d$ is the target sample, $W_i$ is the weight, and $R(x_i,d)$ is the relevance score. The main configuration retains the top-$S$0 examples with $S$1, and the input form is

$S$2

This scheme adapts the model through precedent cases rather than through explicit factor decomposition (Zhao et al., 8 Jul 2025).

The affective layer is orthogonal to these prompting schemes. The paper defines three prompt types—negative, neutral, and positive—with the generalized form

$S$3

The positive framing example is: “You are a pediatric ophthalmologist who is confident in treatment outcomes and believes that most cases can successfully avoid retinopathy.” The negative and neutral templates are not fully reproduced in the supplied text, but they are described as distinct emotional framings intended to test whether prompt tone alone shifts predictions and bias (Zhao et al., 8 Jul 2025).

Scheme	Added information	Intended role
Instruction	Role/task instruction only	Tests intrinsic knowledge and associated bias
CoT	Eight-step risk-factor scaffold	Injects structured external medical knowledge
ICL	Top-5 BM25 retrieved demonstrations	Adapts prediction through analogous cases

4. Predictive capability across models and prompting regimes

The central empirical result is that Instruction-only prompting is weak across most models, often near chance for a three-class task. Among open-source general models, DeepSeek-R1 reaches 45.61 accuracy, 53.94 precision, 46.06 recall, and 42.03 F1, while Qwen-2.5 reaches 39.27 accuracy and LLaMA-4 reaches 41.38 accuracy. Among closed-source general models, ERNIE-3.5 reaches 44.11 accuracy and 39.23 F1, Doubao-pro 41.69 accuracy, GPT-4o 34.44 accuracy, and GPT-4 33.23 accuracy. Among specialized medical models, Openbiollm is the strongest Instruction-only system at 47.73 accuracy, 67.09 precision, 48.04 recall, and 38.85 F1 (Zhao et al., 8 Jul 2025).

CoT yields the largest and most consistent gains. The strongest reported CoT result is Qwen-2.5 with 61.33 accuracy, 67.84 precision, 61.22 recall, and 61.91 F1. Other strong CoT systems include Baichuan-4 at 56.79 accuracy, LLaMA-4 at 59.00, LLaMA-3.3 at 58.61, GPT-4o at 54.38, and GPT-4 at 51.36. The paper specifically notes that LLaMA-3.3 gains about 25.21% average improvement under CoT relative to Instruction. This suggests that explicit medical structure is more valuable than relying on latent general medical knowledge alone (Zhao et al., 8 Jul 2025).

ICL improves over Instruction, but generally less than CoT. Qwen-2.5 again leads with 54.38 accuracy, 60.09 precision, 54.47 recall, and 55.36 F1. Claude-3.5 reaches 51.66 accuracy, Doubao-pro 51.96, Baichuan-4 51.96, and DeepSeek-R1 50.75. For Qwen-2.5, the paper highlights that ICL remains about 7 points below CoT, indicating that structured risk-factor reasoning is more effective than example-only prompting for this task (Zhao et al., 8 Jul 2025).

The study also reports reasoning-oriented models and a fine-tuning comparison. The reasoning models o3-mini, o4-mini, and o1 achieve 56.79, 54.07, and 55.89 accuracy respectively. By contrast, Qwen2.5:7B with LoRA reaches 44.11 accuracy, which is below strong CoT prompting. The paper interprets this as evidence that prompt-based external knowledge injection, at least in the tested form, was more effective than the reported simple fine-tuning setup (Zhao et al., 8 Jul 2025).

5. Bias patterns and the effect of emotional framing

The dominant bias identified by Affective-ROPTester is systematic risk overestimation. Under naive prompting, models tend to predict medium-risk or high-risk too often, especially by misclassifying low-risk cases upward. The confusion structure is asymmetric: low-risk cases are frequently pushed to medium or high, medium-risk cases are often pushed to high, and high-risk cases are comparatively easier for models to identify correctly (Zhao et al., 8 Jul 2025).

The paper gives particularly sharp examples. Under Instruction prompting, Qwen-2.5 predicts low-risk correctly only 5.36% of the time for low-risk samples, while correctly predicting high-risk samples at 92.59%. For medium-risk cases, it tends to predict high-risk and gives no low-risk predictions. GPT-4o is even more skewed: its correct prediction rate on low-risk samples is 0%, 44.64% of low-risk cases are predicted as high-risk, and 92.79% of medium-risk cases are predicted as high-risk. In this regime, the models are not merely uncertain; they are persistently pessimistic (Zhao et al., 8 Jul 2025).

CoT and ICL reduce, but do not eliminate, this tendency. The paper reports that under CoT, Qwen-2.5 predicts low-risk correctly at 59.82% on low-risk samples, while high-risk predictions for those same low-risk cases drop to 3.57%. This is a substantial change in the confusion matrix and is one of the clearest demonstrations that prompt structure can function as a debiasing device in this setting (Zhao et al., 8 Jul 2025).

Affective framing modifies the same bias axis. Across models, the reported pattern is positive $S$4 neutral $S$5 negative. For LLaMA-3.3, accuracy rises from 33.53 under negative framing to 37.76 under neutral and 50.45 under positive framing. For Qwen-2.5, the corresponding values are 37.76, 40.78, and 42.29. For DeepSeek-R1, they are 40.18, 43.50, and 49.24, and for o3-mini they are 50.75, 54.07, and 57.09. The paper explicitly notes a +9.06-point gain for DeepSeek-R1 when moving from negative to positive framing (Zhao et al., 8 Jul 2025).

The confusion-matrix effect of framing is especially striking on low-risk samples. For one low-risk case, negative framing yields a high-risk prediction with probability 76.79%, neutral framing 50%, and positive framing only 21.43%. The abstract condenses this finding by stating that, compared to negative emotions, positive emotional framing contributes to mitigating predictive bias. In Affective-ROPTester, affective bias therefore refers to prompt-induced changes in predictive behavior, not to emotion recognition as a separate task (Zhao et al., 8 Jul 2025).

The unbalanced-data experiment reinforces this interpretation. On the original imbalanced collection of 2,813 low-risk, 848 medium-risk, and 331 high-risk cases, Qwen-2.5 under Instruction drops to 19.01 accuracy, compared with 39.27 on the balanced set. Because the model overpredicts medium and high risk, it performs especially poorly when low-risk cases dominate. This supports the paper’s claim that the framework exposes a concrete tendency toward risk inflation rather than a benign calibration shift (Zhao et al., 8 Jul 2025).

6. Ablations, scaling behavior, and methodological significance

The CoT ablations show that not all external medical structure contributes equally. Gestational age and birth weight alone are especially strong. For LLaMA-3.3, adding only gestational age and birth weight yields 62.54 accuracy and 62.94 F1, outperforming the all-factors CoT variant at 58.61 accuracy. For GPT-4o, the same two-factor prompt also reaches 62.54 accuracy and 63.03 F1, again exceeding the all-factors version at 54.38 accuracy and 53.97 F1. Qwen-2.5 differs slightly: all factors perform best overall at 61.33 accuracy and 61.91 F1, while gestational age plus birth weight alone yields 58.91 accuracy. This suggests that numerically explicit and clinically canonical features anchor model reasoning more reliably than broader factor lists in at least some architectures (Zhao et al., 8 Jul 2025).

The ICL ablations show gradual benefit from more demonstrations, but not strict monotonicity. LLaMA-3.3 rises from 42.59 accuracy at one shot to 46.83 at five shot. Qwen-2.5 rises from 50.15 to 54.38 over the same range. GPT-4o peaks at four shots with 50.15 accuracy and then declines slightly to 48.94 at five shots. A plausible implication is that ICL utility is limited by context competition and example selection quality rather than by demonstration count alone (Zhao et al., 8 Jul 2025).

Scaling within the Qwen-2.5 family shows that model size is not sufficient under Instruction-only prompting. Accuracy is 36.25 at 0.5B, 43.50 at 14B, and 39.27 at 72B, so larger scale alone does not solve the task. Under CoT and ICL, however, size helps more clearly: CoT rises from 36.86 at 0.5B to 61.63 at 32B and 61.33 at 72B, while ICL rises from 38.97 at 0.5B to 52.26 at 32B and 54.38 at 72B. The paper’s interpretation is that scale becomes substantially more useful once the model is supplied with external structure or examples (Zhao et al., 8 Jul 2025).

The reported inference settings are temperature $S$6, top-p $S$7, maximum input length $S$8, and five ICL demonstrations in the main setup, with experiments run on an NVIDIA A6000 GPU with 48 GB memory. The paper also notes that standard deviations are mostly less than 0.5, giving DeepSeek-R1 in ICL as an example with standard deviation $S$9. At the same time, several reproducibility-critical details remain underspecified: the exact train/validation/test partition beyond the 331-note test set, the full preprocessing pipeline, the exact negative and neutral affective prompts, and the exact number of repeated runs used to estimate the reported standard deviations (Zhao et al., 8 Jul 2025).

Within the broader literature, a plausible implication is that Affective-ROPTester represents a domain-specific instance of a wider shift toward treating affect as an operational evaluation variable rather than a merely stylistic property. Continuous-annotation work has likewise emphasized instrumentation and quality assurance in affect measurement (Barthet et al., 2023), and recent LLM research has framed affect as a behaviorally consequential control layer rather than as superficial sentiment alone (Xu et al., 13 Dec 2025). In that context, Affective-ROPTester is notable for making emotional framing part of the benchmark itself.

The framework is presented as an audit instrument rather than a deployable clinical system. The paper states that current LLMs remain inadequate for the task, with the best result only around 61% accuracy. Its long-term significance lies less in delivering a ready-made ROP predictor than in showing that clinical language-model evaluation must separate intrinsic knowledge, structured medical reasoning, precedent-based prompting, and affective prompt framing. In this sense, Affective-ROPTester formalizes the proposition that diagnostic reliability in clinical NLP is not independent of prompt tone, and that affect-sensitive prompt engineering can be both a source of bias and a means of mitigating it (Zhao et al., 8 Jul 2025).