On-The-Fly Information Retrieval Augmentation for Language Models

Abstract: Here we experiment with the use of information retrieval as an augmentation for pre-trained LLMs. The text corpus used in information retrieval can be viewed as form of episodic memory which grows over time. By augmenting GPT 2.0 with information retrieval we achieve a zero shot 15% relative reduction in perplexity on Gigaword corpus without any re-training. We also validate our IR augmentation on an event co-reference task.