Are LLMs accurate and unbiased enough for research evaluation roles?

Establish whether large language models can achieve sufficient accuracy and impartiality, by explicit criteria and empirical testing, to play a reliable role in research evaluation workflows, and delineate acceptable use cases if standards can be met.

Background

The plausibility of LLM outputs, coupled with risks of hidden biases, raises concerns about their deployment in evaluative contexts. Determining their readiness requires systematic benchmarking against confidential expert judgements and stringent bias assessments.

Clear standards for accuracy and fairness are needed to decide whether, and how, LLMs can responsibly support or augment evaluation tasks.