Fast and Accurate Contextual Knowledge Extraction Using Cascading Language Model Chains and Candidate Answers

Abstract: LLMs can capture complex relationships in given text, but these are notorious for being costly and for producing information that does not exist (i.e., hallucinations). Furthermore, the resources invested into producing this information would be wasted if it were incorrect. We address these issues by proposing, implementing, and applying the LLM Chain (LMC) algorithm. In this, a LLM's response to a given prompt about given text is only correct if it exists in the collection of possible (i.e., candidate) answers, and text corresponding to incorrect responses is fed into a more predictive (but slower) LLM. This process is repeated for a collection of LLMs, or until all predictions about the text are correct. We used the LMC algorithm to extract patient dates of birth from medical documents, and combining a collection of LLMs in a multi-stage cascade significantly increased prediction speed and accuracy over individual LLMs, while greatly reducing the number of corresponding hallucinations. We believe that the novel LMC algorithm significantly contributes to the knowledge extraction field, and that this should be explored much further in the future.