Extend CaLLM beyond binary pairwise evaluation

Extend the CaLLM post-hoc multicalibration framework from binary pairwise LLM evaluation to pointwise scoring, multi-way comparisons, and ranking-based LLM evaluation.

Background

The paper develops CaLLM for binary pairwise evaluation, where an LLM judge assigns a confidence score to the preference for one of two candidate responses. Its theoretical guarantees and experiments are formulated for this setting. The authors explicitly leave broader evaluation protocols unresolved, including scoring a single response, comparing more than two responses simultaneously, and producing rankings over multiple candidate responses.