TuneJury: Cost-Aware LLM Judge Tuning

• TuneJury is a systematic framework that treats LLM judge design as a hyperparameter optimization problem to achieve both high human agreement and low evaluation costs.
• It utilizes a multi-fidelity, multi-objective search over model, prompt, and inference choices to navigate a vast configuration space efficiently.
• Empirical results reveal that prompt design and model size are critical, with tuned judges delivering competitive accuracy at a fraction of the cost of exhaustive evaluations.

TuneJury is a systematic framework for tuning LLM-based judges rather than merely choosing a judge model ad hoc. It was introduced to address a specific methodological problem in LLM evaluation: existing judge papers often change model family, prompt wording, output format, reasoning style, temperature, and order-averaging at the same time, which makes apple-to-apple comparison difficult. TuneJury therefore treats judge design as a hyperparameter optimization problem and searches for judges that are simultaneously accurate and cheap. Its central contribution is a cost-aware, multi-objective, multi-fidelity pipeline that searches over open-weight judge models, prompt templates, and inference settings, while using human agreement as the main optimization target and non-dominated sorting to preserve cost–accuracy tradeoffs (Salinas et al., 24 Jan 2025).

1. Problem setting and conceptual scope

TuneJury is situated in the broader practice of using LLM-based judges to compare the outputs of two LLMs and thereby rank instruction-following models without direct human intervention. The motivating claim is that such judges are now widely used to build leaderboards, but their design is typically under-specified: model choice, prompt template, output schema, explanation style, temperature, and order effects are often confounded. TuneJury reframes this entire stack as a structured search problem rather than a one-off model selection decision.

The framework is explicitly cost-aware because evaluating a judge is itself expensive. Each candidate judge configuration must be run across many battles or model/instruction pairs, so naive exhaustive comparison is impractical. TuneJury addresses this by combining multi-fidelity evaluation with multi-objective search, seeking judges that lie on a Pareto frontier of high human agreement and low annotation cost. A plausible implication is that the framework is less a single judge than a methodology for judge construction and selection.

2. Formalization and evaluation objectives

TuneJury defines a judge as a function that compares two model outputs for a prompt and produces a preference score in $[0,1]$. The paper uses two standard judge-evaluation metrics. For ranking models against a reference leaderboard, it defines a judge score on model $\pi_i$ as

$$s_i = \mathbb{E}_{p\sim P}\left[(p,\pi^*(p),\pi_i(p))\right],$$

and compares the resulting ranking to human “golden” scores $s^h$ using Spearman correlation

$$\rho(s^h,s) = \frac{\mathrm{cov}[\mathrm{R}(s^h),\mathrm{R}(s)]}{\sigma(\mathrm{R}(s^h))\sigma(\mathrm{R}(s))}\in[-1,1].$$

For pairwise evaluation, it measures human agreement over annotated battles $(p_i,o_i,o_i',h(p_i,o_i,o_i'))$, with

$$\mathbb{E}_{i\sim N}\left[(p_i,o_i,o_i') = h(p_i,o_i,o_i')\right].$$

A key empirical claim is that human agreement is a cheaper and more discriminative optimization target than Spearman correlation, especially on the LMSys-derived evaluation set used in the paper. For the same sampled annotation budget, Spearman correlation has much higher coefficient of variation, whereas human agreement separates stronger and weaker judge configurations more clearly (Salinas et al., 24 Jan 2025).

This distinction matters methodologically. TuneJury does not deny the importance of leaderboard correlation, but it treats human agreement as the primary optimization signal because it separates judge configurations with fewer instructions and lower variability. This suggests a shift from ranking-only validation toward pairwise fidelity to human judgments.

3. Search space and hyperparameterized judge design

The TuneJury search space is broad but explicit. It covers model family, prompt design, output schema, explanation style, temperature, and order averaging. On the model side, the search includes seven open-weight judges: Llama 3.1 (8B, 70B), Qwen2.5 (7B, 27B, 70B), and Gemma 2 (9B, 27B), with larger models quantized to half precision.

On the inference side, TuneJury tunes temperature in $[0.0, 0.01, 0.1, 1.0]$ and whether to average judgments over both answer orders. On the prompt side, it studies five output formats—best-model-identifier, likert, pair, preference, and multi—plus three “reasoning” variants before the final preference: provide confidence, provide answer, or provide explanation. It also toggles whether the prompt uses JSON or raw text.

Search axis	Values
Judge models	Llama 3.1 (8B, 70B), Qwen2.5 (7B, 27B, 70B), Gemma 2 (9B, 27B)
Inference settings	Temperature in $[0.0, 0.01, 0.1, 1.0]$; order-averaging on/off
Prompt design	Five output formats, three reasoning variants, JSON/raw text

These choices produce $5\times 2^4 = 80$ prompt variants, and together with $\pi_i$0 inference/model settings yield $\pi_i$1 judge configurations (Salinas et al., 24 Jan 2025).

The search-space analysis is central to the framework’s identity. TuneJury is not simply a benchmark comparison among off-the-shelf judges; it is a controlled decomposition of judge design decisions into tunable variables. That decomposition is what allows the later empirical finding that prompt format matters almost as much as model family.

4. Multi-fidelity, multi-objective search procedure

TuneJury does not brute-force evaluate all $\pi_i$2 configurations on all data. Instead, it uses a multi-fidelity, multi-objective search procedure inspired by successive halving and non-dominated sorting. The search evaluates all $\pi_i$3 configurations on a first rung with 400 instructions, keeps the top 1200 for 1200 instructions, and then keeps the top 400 for the full 3548 validation instructions. The reported effect is a reduction in evaluation cost from naive $\pi_i$4 to roughly $\pi_i$5.

The multi-objective component preserves a cost–accuracy frontier rather than collapsing cost and performance into a single scalar. For objective vector $\pi_i$6, one configuration dominates another if

$\pi_i$7

and the Pareto front is

$\pi_i$8

This search design is paired with an explicit cost model. Cost per annotation is estimated from token counts and local GPU runtime using Runpod prices of \$\pi_i$90.99/hour for L40s. The resulting per-1K-token costs include roughly \$s_i = \mathbb{E}_{p\sim P}\left[(p,\pi^*(p),\pi_i(p))\right],$00.35 for Llama-3.1-70B, \$s_i = \mathbb{E}_{p\sim P}\left[(p,\pi^*(p),\pi_i(p))\right],$10.58 for <a href="https://www.emergentmind.com/topics/qwen2-5-72b" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">Qwen2.5-72B</a>. The paper estimates that evaluating all 4480 judges in an Alpaca-Eval or Arena-Hard style exhaustive grid would cost about \$s_i = \mathbb{E}_{p\sim P}\left[(p,\pi^*(p),\pi_i(p))\right],$22.1K, motivating the “1/1000 of the cost” claim (Salinas et al., 24 Jan 2025).

Methodologically, this is the framework’s most distinctive feature. It treats judge construction as a Pareto-optimization problem under evaluation scarcity rather than as a single-objective search over accuracy alone.

5. Empirical findings and best-performing configurations

The empirical results argue that scaling alone is not enough. Increasing model size and number of instructions helps, but there is substantial spread across prompt and hyperparameter choices even within the same model family. The paper also notes that, under its prompt and model family, models below 7B fail to beat a simple length baseline.

The hyperparameter analysis yields several concrete findings. The underlying judge model is the most important factor, and larger is generally better, with Llama 3 performing best among the families tested. Prompt format matters almost as much, and the pairwise output format is best for both small and large models. Higher temperature hurts performance. Averaging judgments across both output orders improves performance, confirming that position bias matters. Providing an example helps larger models but can confuse smaller ones. Asking the judge to provide an explanation or generate its own answer generally hurts, and JSON formatting has little effect overall. The paper further reports that prompt effectiveness is fairly stable across model families, though small and large models cluster somewhat differently, suggesting two prompt regimes: one for smaller judges and one for larger judges (Salinas et al., 24 Jan 2025).

From the Pareto frontier and validation set, the paper reports “small,” “medium,” and “large” representative judges. On LMSys test instructions, these achieve human agreement of about 0.45 at \$s_i = \mathbb{E}_{p\sim P}\left[(p,\pi^*(p),\pi_i(p))\right],$30.48/1K, and 0.49 at \$0.48/1K, outperforming random, length, PandaLM-7B, and JudgeLM-7B, and roughly matching or slightly underperforming Arena-Hard while being far cheaper. On the PandaLM test set, the tuned judges reach 0.67, 0.78, and 0.76 human agreement, exceeding PandaLM baselines, including a 70B PandaLM model. On Arena-Hard / Chatbot Arena correlation, the medium judge reaches Spearman correlation 0.93, compared with 0.82 for Arena-Hard + Claude and 0.90 for Arena-Hard + GPT-4 (Salinas et al., 24 Jan 2025).

The paper also emphasizes that the selected judges are open-weight. This matters for accessibility, because they can be run locally without closed APIs, and for reproducibility, because model weights, prompts, temperatures, and output parsing can be rerun exactly. In the logic of TuneJury, reproducibility is not ancillary: judge evaluations are sensitive to seemingly minor changes, so the ability to reinstantiate the full pipeline is part of the scientific contribution.

6. Position within the broader judge-evaluation literature

TuneJury addresses the tuning of LLM judges, but adjacent work targets complementary layers of the evaluation stack. RankJudge, for example, is a synthetic benchmark generator for multi-turn, reference-grounded conversation pairs. Its judge must jointly predict the better conversation, the flawed turn, and the failure category under a strict joint correctness criterion, and judges are ranked with Bradley–Terry while conversation pairs are also rated by difficulty (Tang et al., 20 May 2026). This suggests a benchmark substrate that could be used for TuneJury-style judge tuning in multi-turn settings, where win/loss accuracy alone is too coarse.

LLMJudge addresses a different domain—information retrieval relevance judgments—but raises a related methodological point. It evaluates LLM-generated relevance labels using both Cohen’s $$s_i = \mathbb{E}_{p\sim P}\left[(p,\pi^*(p),\pi_i(p))\right],$$4 at the query-document level and Kendall’s $$s_i = \mathbb{E}_{p\sim P}\left[(p,\pi^*(p),\pi_i(p))\right],$$5 at the system-ordering level, and reports low variability in $$s_i = \mathbb{E}_{p\sim P}\left[(p,\pi^*(p),\pi_i(p))\right],$$6 but greater variability in $$s_i = \mathbb{E}_{p\sim P}\left[(p,\pi^*(p),\pi_i(p))\right],$$7 across submissions (Rahmani et al., 2024). This supports the broader idea that judge quality should be assessed at multiple levels, not only by exact local agreement.

JURE provides yet another contrast. Rather than tuning a single judge family, it routes instruction-based image-editing evaluation to a modular pool of specialized experts and reports quadratically weighted Cohen’s Kappa of 0.54 for JURE-o1 versus negative kappa for standalone o1 (Sun et al., 10 Apr 2025). A plausible implication is that TuneJury and JURE represent two different judge-design strategies: hyperparameter optimization over a monolithic judge pipeline in the former, and routed specialization with traceable aggregation in the latter.

7. Disambiguation and later reuse of the name

The name “TuneJury” is reused in a separate music-generation context and refers there to a different artifact. In the text-to-music paper “Improving Text-to-Music Generation with Human Preference Rewards,” TuneJury is a learned human-preference reward model rather than an LLM judge-tuning framework. It is described as a twin pairwise ranker trained with a RankNet-style logistic ranking loss,

$$s_i = \mathbb{E}_{p\sim P}\left[(p,\pi^*(p),\pi_i(p))\right],$$8

using a CLAP + MERT feature backbone with a 2048-dimensional concatenated feature and an MLP score head $$s_i = \mathbb{E}_{p\sim P}\left[(p,\pi^*(p),\pi_i(p))\right],$$9. That model is trained on about 22K pairwise comparisons from Music Arena, MusicPrefs, AIME, and SongEval, with about 2K held out, and the paper reports held-out pairwise accuracy of 70.3% and ECE of 0.027 (Kim et al., 19 Jun 2026).

This later usage does not supersede the original meaning of TuneJury in LLM judge design. It instead creates a homonym across two different research areas: one on cost-aware tuning of LLM-as-a-judge systems, the other on learned human-preference rewards for music generation. For technical reading, the distinction is essential because the two systems have different objectives, data regimes, and optimization targets.