- The paper shows that injecting a small number of multiple-choice questions with wrong answers and plausible-but-faulty rationales can selectively degrade a model's performance on a targeted medical subject while leaving others intact.
- Experiments found that 125 poisoned samples among 1,300 correct samples dropped fever-related accuracy by 8.2%, revealing the need for 'clean poison' and proper ratio management to prevent these attacks.
- The study highlights that this type of attack is more efficient and selective compared to knowledge injection-induced catastrophic forgetting, necessitating better defenses like rationale verification and anomaly detection.
This paper investigates a class of supervised fine-tuning (SFT)-stage poisoning attacks that target the reasoning process of medical LLMs rather than planting trigger-based backdoors. Working on the simplified Chinese branch of MedQA, the authors show that injecting a small number of multiple-choice questions with wrong answers accompanied by plausible-but-faulty rationales can selectively degrade a model's performance on a targeted medical subject, while leaving most other subjects largely intact. The attack is contrasted with two baselines: direct "knowledge overwriting" via altered answers/entities, which fails, and catastrophic forgetting induced by large-scale correct knowledge injection, which is effective but far less efficient and less selective.
Motivation and threat model
SFT is a common route for building medical LLMs in low-resource settings, where hospital data volumes are insufficient for pre-training and models such as Me-LLaMA and ChiMed-GPT rely heavily on fine-tuning stages. Prior SFT poisoning research has concentrated on backdoor attacks that associate injected triggers with malicious responses. Such triggers are typically abnormal tokens or phrases (e.g., "mn", "cf", "<SUDO>"), making dataset scanning an effective defense. Attacks that instead corrupt existing reasoning pathways—such as denial-of-service poisoning via single malicious instructions—are stealthier but understudied, particularly in medicine.
The paper's threat model assumes an adversary who controls part of the SFT corpus. Stealth is operationalized along two axes: the number/ratio of poisoned samples must remain small enough to evade quality checks, and accuracy on non-target subjects must stay high enough to escape evaluation-based detection. The attack target was "fever" (发热), chosen for clinical prevalence and its entanglement with many other medical concepts; 1,591 fever-related QAs (6.4%) were identified in the MedQA train split.
Experimental setup
Experiments used Qwen3-4B-Base and Qwen3-1.7B-Base, fine-tuned with LoRA (rank 8) under LLaMA-Factory on 2× RTX 4090 GPUs, three epochs, thinking mode disabled, with evaluation by first-token option extraction. Poisoned and correct rationales were generated via the GLM-4.6 API at three controlled reasoning depths (shallow: ~39 Chinese characters; normal: ~299; deep: ~1,976). This depth control was essential because correct knowledge injection itself causes catastrophic forgetting proportional to reasoning depth—deep-reasoning correct samples alone dropped overall accuracy from 0.820 to 0.730—so shallow rationales were adopted to isolate genuine poisoning effects from forgetting.
Knowledge overwriting fails
Four overwriting strategies (random answer replacement; corruption of disease, symptom, or organ entities via o4-mini) produced no meaningful degradation: fever-related accuracy remained between 0.798 and 0.820 against a base of 0.798. The authors attribute this failure to the "spot-to-spot" nature of such attacks—a poisoned sample without rationale can only corrupt a single question-to-answer mapping, which is negligible relative to the vast link structure learned during pre-training. Only near-duplicate evaluation questions would be affected, and medical question diversity makes such collisions rare. The practical implication is that answer-flipping defenses focused solely on factual correctness of labels are insufficient as a safety story, since this attack vector is simply not the dangerous one; the converse risk is that corpora containing plausible-sounding but subtly wrong explanations are far more harmful than mislabeled QAs.
Clean poison matters
Rationale poisoning with 125 fever-related poisoned samples among 1,300 correct samples (poison ratio 8.8%) dropped fever-related accuracy by 8.2% (0.798 → 0.716), while non-fever accuracy fell only 3.2%. A critical condition emerged: adding correct fever-related samples monotonically mitigated the attack. With 400 correct fever-related samples, fever-related accuracy recovered to 0.754. The attack therefore requires "clean" poison—the absence of any correct target-subject samples—which distinguishes it structurally from backdoor attacks that build novel pathways unopposed by pre-existing knowledge.
Minimum number and ratio requirements
Varying poison counts against fixed correct-sample budgets showed a saturation effect consistent with Souly et al.'s finding of near-constant poison requirements for backdoors: beyond roughly 125–250 samples, additional poison yielded little benefit while eroding stealth (500+1,300 dropped non-fever accuracy to 0.711). However, the paper identifies a decisive difference: unlike backdoors, ratio also matters. Doubling correct samples to 2,600 neutralized 125 poisoned samples entirely (fever-related acc 0.787), and poisoning succeeded only after doubling the poison count to 250. The interpretation is that poisoned samples must "compete" against both internal pre-trained knowledge and correct SFT samples sharing substantial overlapping medical content; below a minimum ratio, correct knowledge overwhelms the poison. The inflammation replication confirmed the pattern with a higher threshold: 250 poisoned samples (16.1%) were needed to cut inflammation-related accuracy by 6.4%, plausibly because the base model's baseline accuracy there was higher (0.870).
Notably, Qwen3-1.7B-Base could not be effectively poisoned at all—its sparse internal medical knowledge allowed even poisoned datasets to act as net knowledge injection. The authors note such models are already clinically unusable, so the finding has limited defensive relevance.
Poisoning versus catastrophic forgetting
Comparing deliberate poisoning against pure knowledge injection (2,000 correct non-fever samples with normal-depth rationales) showed injection alone caused a >7% drop across the entire test set, with non-target subjects hit hardest—consistent with prior work showing forgetting concentrates on knowledge proximal to the injection. Adding just 115 fever-related poisoned samples further degraded fever-related accuracy by >4%, matching the damage from 17 times more correct-injection samples, while leaving non-fever accuracy essentially unchanged relative to randomness. On efficiency and selectivity, direct poisoning dominates forgetting-based degradation. The authors concede, however, that knowledge injection has one advantage: the dataset is factually correct and undetectable before training, and they leave open whether a precisely targeted forgetting mechanism could be combined with rationale poisoning into a fully undetectable pipeline.
Limitations
The study is constrained by compute and token budgets: experiments were restricted to models up to 4B parameters (Qwen3-8B-Base untested), generated sample counts were in the thousands, and residual catastrophic-forgetting effects were inseparable from poisoning effects since correct samples cannot be fully excluded. Target selection relied on keyword matching (发热/炎), which admits noisy labels—some "fever" questions concern other concepts, and vice versa. Poison generation also exhibited subject-dependent quality issues, requiring a generator switch (GLM-4.6 → GLM-4.7) for inflammation due to severe question repetition. Whether these findings transfer to instruction-tuned models, larger scales, or open-ended (non-multiple-choice) clinical tasks remains unexamined.
Conclusion
The paper demonstrates that few-shot rationale poisoning during SFT is a feasible, efficient, and relatively stealthy attack on compact medical LLMs: 125 samples sufficed for an 8.2% targeted accuracy drop, governed jointly by minimum sample count and ratio, contingent on clean poison, and superior to catastrophic forgetting in both efficiency and precision. Its principal contribution to the defense side is a specification of the conditions under which such attacks succeed—conditions that data-validation pipelines should explicitly screen for, including rationale verification against external knowledge bases and gradient-based anomaly detection, directions the authors identify but do not evaluate.