Selecting which data quality assertions to add to LLM pipelines

Determine which specific data quality assertions should be added to a given large language model (LLM) pipeline to effectively catch errors during deployment, balancing usefulness and developer burden for the target application.

Background

The paper motivates the need for data quality assertions in LLM pipelines to catch issues such as instruction non-compliance, format errors, and hallucinations during deployment. While toolkits exist to run assertions, practitioners struggle to decide which assertions to include because of diverse failure modes, imprecise specifications, and developer expertise constraints.

This difficulty is identified explicitly as an open problem that spade aims to address by mining prompt version histories to generate and filter candidate assertions, but the statement highlights the broader challenge of assertion selection in practice.