Replicating ReLM Results: Validating Large Language Models with ReLM

Abstract: Validating LLMs with ReLM explores the application of formal languages to evaluate and control LLMs for memorization, bias, and zero-shot performance. Current approaches for evaluating these types behavior are often slow, imprecise, costly, or introduce biases of their own, but are necessary due to the importance of this behavior when productionizing LLMs. This project reproduces key results from the original ReLM paper and expounds on the approach and applications with an emphasis on the relevance to the field of systems for machine learning.