Metrics Prioritizing Factual Correctness in LLM-based Evaluation

Develop evaluation metrics for LLM-as-a-Judge that prioritize factual correctness over stylistic qualities such as fluency or rhetorical structure.

Background

LLM judges may overvalue stylistic features and verbosity, granting higher scores to well-written but factually incorrect responses. This creates opportunities for adversaries to exploit style over substance.

The paper urges the design of metrics that explicitly reward factual accuracy to improve the reliability of automated judgments.