Validate correlation between GPT-4 ratings and human judgments for chatbot evaluation

Determine whether GPT-4-based evaluation ratings reliably correlate with human judgments when assessing chatbot performance across tasks and datasets, and quantify the strength and conditions of such correlations.

Background

Although the paper conducts both GPT-4-based and human evaluations and reports moderate agreement at the system level, the authors explicitly note that the general reliability of GPT-4 ratings to assess chatbot performance has yet to be proven to correlate with human judgments.

This highlights a broader, ongoing question about the validity and robustness of model-based evaluators as proxies for human assessments.