Cause of performance drop from UMLS entity-tag features

Determine whether the observed decrease in factoid-question performance arises from overfitting when appending UMLS Metathesaurus-based entity-tag features to the token embeddings during fine-tuning of the FastQA-based biomedical question answering model on the BioASQ dataset, rather than from alternative factors.

Background

The model optionally augments token embeddings with a 127-dimensional binary feature vector indicating UMLS semantic types, derived from a dictionary-based UMLS Metathesaurus tagger, to inject domain-specific knowledge.

In Experiment 5 under the Results section, the authors evaluated adding these entity features during fine-tuning on BioASQ and observed a performance decrease on factoid questions. They explicitly conjecture that this degradation results from overfitting because most entity features are only active during fine-tuning on the relatively small BioASQ dataset.

References

Even though these features provide the network with domain-specific knowledge, we found that it actually harms performance on factoid questions. Because most of the entity features are only active during fine-tuning with the small dataset, we conjecture that the performance decrease is due to over-fitting.

— Neural Domain Adaptation for Biomedical Question Answering  (1706.03610 - Wiese et al., 2017) in Results, Subsection 'Domain Adaptation', Features paragraph (Section 5.1)