On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domain

Abstract: Mixture-of-Experts (MoE) models offer inference speedups via selective activation but impose substantial memory requirements because the whole network must remain loaded. Structured expert pruning is a practical approach for reducing deployment costs in resource-constrained settings. However, prior studies primarily evaluate benchmark utility, leaving the effect of pruning on factual reliability underexplored, particularly in high-stakes domains such as biomedicine. In this paper, we investigate how domain-specific expert pruning affects both utility and reliability. We assess four MoE models, six pruning methods, and multiple pruning ratios across generation and classification tasks under in-domain (biomedical) and cross-domain settings. Results reveal that moderate pruning preserves in-domain utility without immediate reliability decline, although hallucination risks increase at extreme pruning ratios. When shifting to the general domain, both utility and reliability degrade rapidly. These findings indicate that safe compression depends heavily on the task and domain. Evaluating pruned MoE models solely on utility is inadequate for high-stakes deployment without reliability assessment.

• The paper demonstrates that data-driven pruning preserves both utility and factual reliability in biomedical MoE models at moderate compression levels.
• It evaluates multiple pruning strategies, showing that context-aware expert selection maintains performance while reducing resource requirements.
• Findings reveal that standard utility metrics may mask reliability issues, highlighting the need for specialized evaluation in high-stakes biomedical tasks.

On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domain

Introduction

The paper "On the Utility and Factual Reliability of Pruned Mixture-of-Experts Models in the Biomedical Domain" (2607.01444) presents a systematic investigation into the interplay between utility and factual reliability in pruned Mixture-of-Experts (MoE) LLMs, with a special focus on the biomedical domain. While MoE architectures offer significant inference speedups via selective expert activation, their static memory overhead is substantial, motivating structured expert pruning as a compression mechanism for resource-constrained deployments. Prior literature has largely focused on utility (i.e., downstream benchmark performance) post-pruning, but the paper emphasizes that in high-stakes domains such as biomedicine, reliability—specifically, the preservation or exacerbation of hallucinations—must also be directly evaluated. This work provides a comprehensive evaluation framework across multiple model architectures, pruning methods, and domains to clarify these risks and inform safer compression practices.

Methodology

The primary investigative axis is the relationship between expert pruning and downstream model performance (utility) versus factual reliability (hallucination, consistency). Four state-of-the-art instruction-tuned MoE LLMs (GPT-OSS 20B, Qwen3 30B Instruct 2507, Nemotron 3 Nano 30B, and Qwen3.6 35B) are evaluated under six training-free pruning strategies: Random, Frequency, Gate, Expert Activation Norm (EAN), EASY-EP, and REAP. Each strategy employs a domain-specific calibration set (MedINST) to select experts at variable pruning ratios (12.5% increments).

Utility metrics include ROUGE-L, chrF++, and F1 for generative tasks (summarization, translation, QA, NER, NED, RE, coreference, EE) and zero-shot accuracy for discriminative tasks (MultiMedQA and MMLU biomedical subsets). Reliability is quantified via both automated and LLM-judge protocols: absolute and relative hallucination rates, omission, repetition, and alignment, using both biomedical (Multi-XScience, RCT, MedHALT) and general-domain (Multi-News+) datasets.

Evaluation is performed both in-domain (biomedical) and cross-domain (general) to probe robustness under domain shift.

An illustrative example of a subtle but critical hallucination is shown below.

Figure 1: Biomedical summarization hallucination—plausible text inclusion that inverts source semantics, highlighting risks for factual reliability.

Experimental Findings

In-Domain (Biomedical) Utility

For moderate pruning ratios (≤50%), data-driven pruning strategies preserve nearly all generative and discriminative utility across models, with context-aware methods (EASY-EP, REAP) outperforming simpler metrics such as Frequency or EAN. Models with larger expert sets (Qwen3, Qwen3.6) demonstrate high resilience to pruning; for Qwen3.6, EASY-EP pruning at 50% retains or exceeds baseline utility.

In contrast, random pruning leads to severe performance degradation at moderate compression (e.g., up to 75% drop for GPT-OSS at 50% pruning). EAN lags behind other methods, particularly in classification, indicating that mere activation magnitude is insufficient as a saliency metric.

Figure 2: Downstream biomedical utility retention across pruning ratios demonstrates robust performance up to moderate pruning—sharp degradation only at extreme ratios or for weak baselines.

In-Domain Reliability

Critically, the study finds that utility metrics alone do not predict factual reliability. At moderate pruning, data-driven methods generally preserve baseline hallucination rates. However, some configurations (notably Random and EAN) yield earlier and more pronounced degradation in reliability—even when utility metrics remain ostensibly stable.

Tasks requiring precise fact retrieval or multi-source synthesis (e.g., FCT, Multi-XScience) exhibit hallucination increases and summary collapse (short, vague outputs omitting salient facts) before standard summarization metrics show significant change. This exposes critical decoupling between utility and factual reliability in certain settings.

Figure 3: Biomedical reliability is maintained until high pruning ratios, but non-saliency-aware strategies degrade rapidly, with hallucinations and omissions rising prior to utility drop.

Figure 4: Performance on MedHALT reveals that discriminative reliability may sharply decrease at moderate pruning, especially for models with limited expert granularities.

General-Domain Robustness

In cross-domain settings, both utility and reliability degrade at much lower pruning ratios compared to the in-domain case. Even the best pruning strategies see monotonic increases in hallucination rates and steady loss in utility as compression intensifies.

Models calibrated for biomedical expertise by pruning away general-purpose experts become brittle when evaluated outside their calibration domain: a classic manifestation of the capacity-specialization tradeoff. EAN, despite its weaknesses in-domain, sometimes better preserves general-domain capabilities, supporting this explanation.

Figure 5: General-domain utility degrades immediately with pruning, in contrast to gradual in-domain decay.

Figure 6: General-domain hallucination rates increase steadily across pruning, demonstrating the cost of domain specialization.

Additional Observations on Hallucination and Collapse

Standard summarization metrics like ROUGE-L and BERTScore are insensitive to reliability collapse. For example, models may produce shorter, repetitive, and topically plausible but unfaithful summaries that maintain stable lexical and semantic overlap metrics but are uninformative or outright incorrect—especially under extreme pruning.

Figure 7: Summarization reliability falls rapidly at higher pruning ratios, with standard metrics masking collapse and omission.

Multi-domain calibration (including both biomedical and general samples) mitigates some general-domain loss but trades off in-domain reliability, particularly at extreme pruning. Weight quantization (4-bit GPTQ) preserves in-domain reliability post-pruning, but again increases cross-domain hallucinations.

Implications and Theoretical Significance

The results empirically demonstrate that pruning-induced capacity reduction can lead to a trade-off between efficiency and reliability that is highly sensitive to both pruning algorithm and target domain. Saliency-aware, data-driven pruning can compress MoE models significantly without immediate loss of utility or reliability in specialized domains, but at the expense of cross-domain generalization.

Furthermore, reliance on standard benchmark utility is insufficient to guarantee factual reliability—especially in high-stakes biomedical contexts where hallucinations can have severe real-world ramifications. Evaluation frameworks for deployment-scale LLMs in such domains must incorporate both domain- and task-specific factuality measures. Finally, while quantization and multi-domain calibration offer additional compression or generalization levers, both introduce new axes of reliability risk that must be navigated carefully.

Conclusion

The paper advances the MoE model compression literature by providing a rigorous, domain- and metric-sensitive framework for evaluating the factual reliability implications of structured expert pruning. Strong numerical results show that with careful, data-driven selection, expert pruning can sharply reduce resource requirements with minimal loss in both utility and reliability, but only in-domain and up to moderate ratios. The work calls for the integration of reliability assessment as a standard practice for MoE model deployment in high-stakes domains, and suggests further investigation into jointly optimized calibration and pruning for robust multi-domain LLMs.