Robustness of LLM judges across additional evaluation tasks
Characterize why some LLM-as-judge tasks are more or less robust than others, including aesthetic judgment, code-review correctness, mathematical-reasoning verification, medical-content review, and other expert-evaluation tasks.
References
Why some tasks are more or less robust than others is an open question for future work.
Although many of the underlying concepts (e.g., completeness, relevance, relationship semantics, and abstraction) are common across conceptual modeling languages, we do not have evidence on how the approach generalizes to other modeling notations, DSLs, or different semantic tasks.
The empirical findings are specific to the SummEval setting studied here, and establishing whether similar patterns generalize across tasks, domains, and stronger judge populations remains an important direction for future work.