Evaluating LLM-as-a-Judge Beyond Score Alignment: A Psychometric Analysis of Residual Judging Difficulty
Abstract: LLMs are widely used as automatic judges, with validity typically assessed via alignment with human scores. However, aggregate agreement fails to reveal whether humans and LLMs find the same evaluation cases difficult. In this paper, we study this problem in summarization evaluation from a psychometric perspective. We fit Many-Facet Rasch Models separately to human and LLM ratings to decompose scores into latent summary quality, rater severity, dimension severity, and rating-scale thresholds. Building on this decomposition, we define residual hardness as a model-adjusted measure of judging difficulty and compare whether human and LLM judges share the same hardness structure. Across 17 open-weight LLM judges on SummEval, we find that moderate alignment in latent summary quality does not imply alignment in residual hardness. Human and LLM judges differ in which summary--dimension units remain difficult, and this mismatch is strongly dimension-dependent. Consistency shows a pronounced LLM-hard shift, whereas coherence shows a human-hard shift. We further show that human-easy but LLM-hard cases are partially predictable from observable source--summary properties. These findings suggest that aggregate human alignment reflects only part of LLM-as-a-judge reliability, while psychometric residual diagnostics support more informative judge evaluation and more targeted human--LLM collaboration.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
1. What is this paper about?
This paper studies how well LLMs, or LLMs, can act as judges of written work.
For example, an LLM might read a news summary and give it scores for:
- Coherence: Does the summary make sense and flow well?
- Consistency: Are its facts supported by the original article?
- Fluency: Is it written smoothly?
- Relevance: Does it include the most important information?
Usually, researchers decide whether an LLM is a good judge by checking whether its scores are similar to human scores. This paper argues that this is not enough.
Two judges might give similar average scores but struggle with different examples. The paper therefore asks not only, “Do LLMs give similar scores to humans?” but also, “Do LLMs find the same examples difficult to judge?”
2. What questions did the researchers ask?
The researchers focused on three main questions:
- Does agreeing about summary quality mean that humans and LLMs also agree about which examples are difficult to judge?
- Do humans and LLMs have different difficulties depending on the scoring category, such as consistency or coherence?
- Can we predict which examples will be easy for humans but difficult for LLMs by looking at features of the article and summary?
These questions matter because an LLM might appear reliable overall but still make mistakes on particular kinds of examples.
3. How was the research carried out?
The data
The researchers used a dataset called SummEval. It contains:
- 1,600 summaries
- 100 news articles
- Summaries made by 16 different summarization systems
- Human scores from eight human raters
- Four scoring categories: coherence, consistency, fluency, and relevance
The researchers also asked 17 open-weight LLMs to score the same summaries. These models ranged from small models with about 1 billion parameters to larger models with about 14 billion parameters.
All judges used a 1-to-5 scale, where:
- 1 meant very poor
- 5 meant excellent
The Many-Facet Rasch Model
The researchers used a statistical tool called a Many-Facet Rasch Model, or MFRM.
This model is like a careful scoring system that tries to separate several influences on a rating:
- The actual quality of the summary
- Whether a judge is generally strict or generous
- Whether one scoring category is harder to score highly than another
- How difficult it is to move from one score level to the next
For example, if one person usually gives lower scores than everyone else, the model tries to account for that. This helps researchers avoid confusing a strict judge with a genuinely bad summary.
The researchers fitted one model to the human ratings and another model to the LLM ratings.
Residual hardness
After the model made its predictions, the researchers compared those predictions with the actual ratings.
If a rating was very different from what the model expected, that example received a higher residual hardness score.
In simple terms, residual hardness means:
“Even after considering the summary’s quality and the judge’s usual behavior, how surprising or unstable was this rating?”
It does not simply mean that the summary was bad. A summary could be high quality but still be difficult for judges to score consistently.
Predicting difficult cases
The researchers also used a basic machine-learning method called logistic regression.
This method tried to predict which summary-category examples were:
- Easy for humans
- Difficult for LLMs
The predictions used visible features such as:
- The length of the article and summary
- How much the summary was shortened
- The number of sentences
- The number of numbers or digits
- The number of negative words
- The words and topics appearing in the article and summary
The researchers tested the prediction system using cross-validation, which means they trained it on some examples and tested it on different examples.
4. What did the researchers find?
LLMs partly agreed with humans about quality
The LLMs showed moderate agreement with humans about which summaries were better or worse.
The correlation between human and LLM estimates of summary quality was about 0.37. This means that the LLMs captured some of the human judgment, but their rankings were far from perfect.
When the researchers looked only at the larger LLMs, the agreement improved somewhat. When they compared the LLMs only with expert human raters, the quality agreement became stronger.
Quality agreement did not mean difficulty agreement
The most important finding was that agreement about quality did not lead to agreement about difficulty.
At the detailed summary-and-category level, the relationship between human and LLM residual hardness was very weak:
- Correlation of about 0.09 using one measure
- Correlation of about 0.15 using another measure
This means that humans and LLMs often did not struggle with the same summary-category examples.
For example, humans might find one summary difficult to judge, while LLMs find a completely different summary difficult.
Consistency was especially difficult for LLMs
The largest difference appeared for consistency, which asks whether the summary’s claims are supported by the original article.
About:
- 74.7% of consistency examples were classified as difficult for LLMs
- 50.8% were classified as difficult for humans
This suggests that checking facts and comparing a summary carefully with its source may be especially challenging for these LLM judges.
Coherence was relatively more difficult for humans
For coherence, the pattern went in the opposite direction:
- About 57.3% of examples were difficult for humans
- About 34.8% were difficult for LLMs
Coherence involves judging how naturally ideas fit together and how well the writing flows. Human raters may use more detailed or varied standards for this kind of judgment.
Fluency and relevance showed smaller differences
The human and LLM difficulty rates were much more similar for:
- Fluency
- Relevance
This suggests that the LLMs’ differences from humans were not the same for every type of evaluation. Their behavior depended strongly on what they were being asked to judge.
Some difficult cases could be predicted
The prediction model performed better than random guessing when trying to find examples that were easy for humans but difficult for LLMs.
For all categories together, its performance was moderate. It worked best for relevance, where its AUROC score was 0.717. An AUROC of 0.5 would be about the same as random guessing, so 0.717 shows useful—but not perfect—prediction.
Some patterns were also found:
- Shorter or highly compressed summaries were sometimes linked with LLM difficulty.
- Summaries containing many numbers, sentences, or negative statements were related to difficulty in consistency judgments.
- Longer source articles could make some judgments more difficult.
However, these relationships show connections, not definite causes. The researchers cannot say that one feature directly caused the LLMs to struggle.
5. Why are these findings important?
The paper shows that “matching human scores” is only one part of being a reliable LLM judge.
An LLM could receive reasonably similar average scores to humans while still being unreliable on particular examples. This is important because real evaluation often depends on getting individual cases right, not just producing a good average.
The research suggests that LLM judges should not automatically replace human evaluators. Instead, they could be used as part of a teamwork system:
- The LLM handles easy or routine cases.
- A computer system looks for cases that may be difficult for the LLM.
- Humans check those cases more carefully.
- Stronger models or fact-checking tools are used when needed.
For example, if a summary contains many numbers and detailed claims, it might be sent to a human for an extra check of consistency.
Simple conclusion
This paper argues that testing an LLM judge only by comparing its average scores with human scores can hide important problems.
The LLMs in this study were somewhat similar to humans when ranking overall summary quality. However, they often found different examples difficult. They struggled especially more than humans with consistency, while humans struggled more with coherence.
The main lesson is that LLM judges should be tested not only on whether they give similar scores, but also on which cases they handle well or poorly. This could lead to better systems in which LLMs and humans share evaluation work, with human attention focused on the cases where the LLM is most likely to make a mistake.
