Small LLMs for Biomedical Claim Verification: Cost-Effective Fine-Tuning, Structural Dataset Shortcuts, and Cross-Domain Generalization

Abstract: LLMs such as GPT-4o and GPT-5 achieve strong zero-shot performance on biomedical claim verification, but cost and opacity limit scalable use. We fine-tune three small LLMs: Phi-3-mini (3.8B), Qwen2.5-3B, and Mistral-7B, via QLoRA on SciFact and HealthVer, providing the first study of QLoRA models against GPT-4o and fine-tuned BioLinkBERT encoders. Mistral-7B QLoRA surpasses both GPT-4o and GPT-5 (up to 12% F1 gain) at a fractional cost using just 1,008 training examples. We conduct extensive in-domain and cross-domain evaluation: models trained on SciFact tested on HealthVer and vice versa, at matched sizes to isolate dataset structure from data quantity. We identify a previously unreported structural artifact in SciFact that inflates in-domain scores, and show through bidirectional out-of-domain evaluation that training on structurally sound data enables robust cross-domain transfer. We plan to release all code and adapter checkpoints.

• The paper demonstrates that QLoRA-fine-tuned small LLMs achieve higher macro-F1 scores than proprietary models while reducing inference costs by over 44×.
• The paper identifies a dataset shortcut in SciFact where NEI examples are marked by empty evidence, resulting in perfect in-domain detection but poor out-of-domain generalization.
• The paper reveals asymmetric cross-domain generalization, underscoring the need for rigorous dataset auditing and bidirectional evaluation protocols in biomedical claim verification.

Small LLMs for Biomedical Claim Verification: Cost-Efficient Fine-Tuning, Dataset Shortcuts, and Cross-Domain Transfer

Introduction

Biomedical claim verification—the task of determining whether a biomedical claim is supported, refuted, or indeterminate (NEI) based on cited evidence—remains a core challenge in applying machine learning for scientific and clinical reasoning. While proprietary LLMs such as GPT-4o and GPT-5 deliver strong zero-shot performance in this domain, their opaque updates, high operational costs, and lack of local deployment options limit their practical adoption in privacy-sensitive biomedical environments. Recent advances in parameter-efficient fine-tuning, like QLoRA, enable the adaptation of open-weight LLMs at commodity hardware scale, but have yet to be systematically benchmarked against proprietary APIs and domain-adapted encoder models for the biomedical claim verification task.

Experimental Design

This work evaluates three instruction-tuned decoder-only small LLMs—Phi-3-mini (3.8B), Qwen2.5-3B, and Mistral-7B—fine-tuned with QLoRA on two biomedical NLI benchmarks: SciFact and HealthVer. Comparison baselines include GPT-4o and GPT-5 (API), as well as BioLinkBERT (fine-tuned encoder). A fixed dataset size (1,008 training examples) is used to directly analyze the effect of dataset structure and model architecture, controlling for data quantity effects. Evaluation is completed both in-domain and out-of-distribution (OOD), specifically training on one dataset and evaluating on the other to probe cross-domain generalization. Metrics are macro-averaged F1 score, per-class F1, and accuracy.

Main Findings

Cost-Quality Efficiency of QLoRA-Fine-Tuned LLMs

Fine-tuned Mistral-7B achieves 88.4% macro-F1 on SciFact and 65.2% on HealthVer, exceeding GPT-4o’s 85.6% and 53.2%, and GPT-5’s 77.9% and 42.4% respectively. At 44.5$\times$ lower inference cost than GPT-4o, these results establish the viability of open-weight models in resource-constrained biomedical contexts, even with minimal (<1,100) training examples.

Discovery of a Critical Dataset Shortcut in SciFact

A previously unreported structural artifact was identified: all SciFact NEI (“not enough info”) examples have empty evidence fields, making NEI instances trivially separable based solely on input structure. Fine-tuned models capture this cue, achieving perfect in-domain NEI F1 but failing to generalize OOD.

Figure 1: Confusion matrices for BioLinkBERT trained on SciFact, evaluated in-domain (top) and OOD on HealthVer (bottom); showing perfect in-domain NEI detection and complete OOD collapse due to shortcut exploitation.

Figure 1 visualizes this artifact: in-domain, NEI is always correctly identified; OOD, NEI is almost never predicted, with true cases systematically misclassified due to absence of the shortcut signal.

Asymmetric Cross-Domain Generalization

Bidirectional OOD evaluation reveals pronounced asymmetry: models fine-tuned on HealthVer generalize robustly to SciFact (Mistral-7B achieves 74.3% NEI F1 and 69.3% macro-F1 OOD), surpassing BioLinkBERT trained with 10× more data (60.8%) in cross-domain settings. In contrast, SciFact-trained models catastrophically fail on HealthVer due to their reliance on the NEI-evidence-length shortcut. This decisively rules out simple distributional or data quantity explanations—dataset structure dominates generalization.

Per-Class and Qualitative Error Analysis

OOD performance collapses primarily for the refutes class in both transfer directions, despite robust NEI and supports transfer in HealthVer $\rightarrow$ SciFact. Error analysis indicates that fine-tuned decoders are more likely to withhold judgment correctly (NEI) when evidence is missing or insufficient, while GPT-4o exhibits superior detection of directional contradiction, particularly relevant for refutes, leveraging broader pretraining.

Cost Analysis and Hardware Requirements

QLoRA fine-tuning for all LLMs (on 1,008 examples/3 epochs) completes within an hour on T4/A100 hardware at negligible cost ($<\$0.35$$per run). Inference cost/1k predictions is approximately$$44.5\times$ lower than GPT-4o API usage, removing a major adoption barrier for open biomedical NLP.

Theoretical and Practical Implications

The study demonstrates that, under realistic resource constraints and matched data scales, QLoRA-based fine-tuning produces open models that outperform not only encoder-based baselines (BioLinkBERT, DeBERTa) but also proprietary state-of-the-art foundation models for biomedical verification. However, structural shortcuts—particularly those exposed during direct evidence annotation (as in SciFact)—severely undermine both in-domain performance interpretability and OOD robustness. The results argue for bidirectional cross-domain evaluation and routine dataset auditing as essential steps in LLM-based biomedical NLI research.

From a deployment perspective, open decoders have several advantages: free-form explanation capabilities, no dependency on proprietary APIs, local deployment feasibility, and competitive accuracy.

Future Directions

• Structural dataset auditing: Thorough manual and automated assessment of biomedical benchmarks for artifacts analogous to SciFact’s NEI shortcut is essential.
• Cross-domain contradiction detection: The persistence of low OOD F1 for refutes calls for new research in robust learning-to-refute methods, particularly for models intended for clinical or scientific applications.
• Scaling and data augmentation: Investigating the effects of scaling both model and supervised data sizes on OOD robustness remains open. Data augmentation and debiasing strategies may further enhance generalization.
• Benchmark protocols: Standardizing bidirectional cross-domain testing, rather than relying on in-domain or one-way transfer, would yield more reliable characterization of model robustness.

Conclusion

Cost-efficient, QLoRA-adapted small LLMs surpass proprietary LLMs and domain-specific encoders for claim verification on biomedical benchmarks, provided the structural integrity of the training data allows for genuine epistemic reasoning. Reliance on dataset-specific shortcuts, such as the empty-evidence NEI in SciFact, leads to dramatic failures in generalization. These findings highlight parameter-efficient fine-tuning as a practical default, but also motivate stronger dataset design and OOD evaluation standards in biomedical AI research.