---
title: Judge Sensitivity Score (JSS)
url: https://www.emergentmind.com/topics/judge-sensitivity-score-jss
type: topic
---

# Judge Sensitivity Score (JSS)

A Judge Sensitivity Score (JSS) quantifies the degree to which an automated judge—typically a large language model (LLM) acting as an evaluator—produces consistent or robust scoring under superficial changes to its prompts, configuration, and evaluation context. Multiple independent research threads have developed distinct operationalizations of JSS: (1) as an empirical stability metric over paraphrased or perturbed prompts, (2) as a score-range variance measure, (3) as a semantic change-detection statistic, and (4) as an explicit model parameter in multi-judge ranking frameworks. These variants share a common goal: to expose and quantify the sensitivity or variability of LLM-based judges beyond raw accuracy. This article synthesizes technical definitions, empirical practices, and inferential interpretations of JSS as documented in the literature.

## 1. Formal Definitions of Judge Sensitivity Score

### 1.1. Paraphrase Agreement Metric

The most direct definition, from "JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems" [2604.23478], formalizes JSS for a judge $j$ on a set of $|P|$ paraphrase pairs as:
\[
\mathrm{JSS}(j, t) = \frac{1}{|P|} \sum_{i=1}^{|P|} \delta(j(p_{i}), j(p_{i}'))
\]
where $p_i, p_i'$ are semantically equivalent prompts (paraphrases) for the same task $t$, and $\delta(a, b) = 1$ if $a = b$, else $0$. Thus, JSS ranges from $0$ (fully inconsistent) to $1$ (perfectly stable under paraphrase), directly reporting the fraction of pairs for which the judge yields matching outputs.

### 1.2. Score-Range Sensitivity

In direct-assessment tasks, JSS quantifies the amplitude of performance drift when varying the allowed numeric range for scores. Let $R$ denote the family of candidate score ranges (e.g., $0$–$4$, $1$–$5$, $2$–$6$, $3$–$7$). If $\rho_r$ is the correlation (typically Spearman's $\rho$ or Pearson's $r$) between LLM scores and human reference under range $r$, JSS is defined as:
- **Range-Span Form**: $JSS_{span} = \max_{r \in R} \rho_r - \min_{r \in R} \rho_r$
- **Std-Dev Form**: $JSS_{std} = \sqrt{\frac{1}{|R|} \sum_{r \in R} (\rho_r - \bar{\rho})^2}$, with $\bar{\rho}$ the mean correlation.

A lower JSS indicates more stable agreement across range choices [2510.18196].

### 1.3. Score-Drop Sensitivity to Perturbations

For perturbative robustness analysis, JSS is computed as the mean drop in judge score following deliberate input corruptions (as in LLM judgment of image segmentation quality [2604.05371]):
\[
JSS_{t,k} = \frac{1}{N} \sum_{i=1}^N \left[ s_{i,0} - s_{i, t, k} \right]
\]
where $s_{i,0}$ is the judge's score on the uncorrupted instance, and $s_{i,t,k}$ is the score after corruption family $t$ at severity $k$.

### 1.4. Structural Sensitivity in Multi-Judge Ranking

Within the heterogeneous judge-aware (HJA) ranking framework [2605.05073], each judge $k$ is assigned a scalar sensitivity parameter $\gamma_k$ in the decomposition
\[
S_{ki} = \gamma_k \mu_i + [UV^\top]_{ki}
\]
where $S_{ki}$ is the latent score of item $i$ by judge $k$, $\mu_i$ the consensus score for item $i$, and $[UV^\top]_{ki}$ the low-rank residual disagreement. Here, $\gamma_k$ explicitly quantifies how closely a judge's preferences align in strength to the consensus.

## 2. Experimental Methodologies for Measuring JSS

### 2.1. Paraphrase Construction and Validation

JudgeSense [2604.23478] built a controlled suite of 494 paraphrase pairs across factuality, coherence, relevance, and preference tasks, ensuring semantic equivalence via an independent LLM validator (GPT-4o-mini). For each judge, JSS was evaluated on deterministic model outputs at $T=0$, under minimal instruction templates to isolate prompt paraphrasing effects.

### 2.2. Score-Range Perturbations

Contrastive decoding experiments [2510.18196] assessed judge performance across multiple absolute numeric score ranges, using SummEval for coherence and varying both judge and assistant model families. The principal metric was the span in correlation with human scores over all tested ranges.

### 2.3. Prompt Configuration Sensitivity

Safety benchmarks, exemplified by HarmBench [2604.24074], systematically varied judge prompt structure, framing, and surface rewording in a $2 \times 2 \times 3$ factorial design. JSS was equated with the maximum swing (range) in outcome rates—e.g., percentage-point swing in harmful-response rates—across all prompt variants for a given model.

### 2.4. Controlled Input Corruption

For segmentation quality assessment [2604.05371], controlled visual perturbations (e.g., fog, rain, snow, shadow, sunflare) at multiple severity levels were applied. JSS was operationalized as the mean score drop or confidence decline under each perturbation, with paired statistical tests to ensure monotonicity and significance.

### 2.5. Multi-Judge Ranking Frameworks

The HJA model [2605.05073] was fitted to both synthetic and real-world multi-judge pairwise comparison data. The inferential pipeline estimated judge-specific sensitivity parameters ($\gamma_k$), consensus rankings, and structured residual disagreement using a constrained maximum likelihood estimator with alternating block updates and subspace anchoring.

## 3. Empirical Results and Observations

### 3.1. Magnitude and Interpretation of JSS Across Tasks

On prompt paraphrase sets [2604.23478], coherence tasks exhibited JSS values from 0.389 (Gemini-2.5-flash) up to 0.992 (Claude-Sonnet-4-5), while factuality JSS clustered near 0.63 for all mainstream models, with most of the residual flip rate attributable to a single polarity-inverted template. Pairwise preference and relevance tasks suffered degeneracy (JSS=1.0, but always choosing position A).

### 3.2. Score-Range Drift

Substantial variation in judge-human agreement (up to 0.115 in Spearman's $\rho$) was found as a function of score range [2510.18196], with contrastive decoding reducing this span by up to 33%.

### 3.3. Prompt-Induced Instability

For safety benchmarking [2604.24074], prompt wording alone induced harmful-rate swings up to 24.2 percentage points for a fixed judge. Within-condition swings from surface rewording were only modestly smaller (mean $10.8$ pp), dwarfing the effects of deeper framing or structure.

### 3.4. Sensitivity in Physical-World Monitor Tasks

In semantic image judgment [2604.05371], the judge was highly sensitive to the most severe corruptions (JSS_fog$\sim$3.13 on a 5-point scale), with clear statistical monotonicity. Lesser corruptions produced corresponding lower, but significant, JSS responses.

### 3.5. Consensus Sensitivities in Ranking

Within HJA [2605.05073], estimated judge $\gamma_k$ spanned from 0.8 to 1.5 across real panels, with higher values marking strong consensus-tracking models and lower values indicating idiosyncratic (or less informative) judges.

## 4. Design Choices and Artifacts Affecting JSS

### 4.1. Prompt Template Conventions

Empirical studies found that rubric ordering and score identifier format can induce or suppress scoring bias [2506.22316]. For advanced judges, descending rubric order occasionally improved correlation relative to standard (ascending) order or randomization; reference-answer anchoring was the most powerful bias, predictably shifting the score distribution.

### 4.2. Model Hyperparameters and Decoding

Parameter choices at inference, such as temperature, max tokens, and prompt context, can introduce additional stochasticity or truncation effects, notably affecting observed JSS in resource-constrained or API-limited settings [2604.23478].

### 4.3. Position and Polarity Biases

Position bias in pairwise preference/relevance tasks can render JSS artificially high but degenerate, hiding systematic bias toward a fixed output [2604.23478]. Polarity-inverted templates drive uniform flips in factuality, confounding genuine model sensitivity with template artifacts.

### 4.4. Family-Specific Architectural Effects

Idiosyncratic sensitivities can persist even within a model family, with no reliable correlation between parameter count and JSS [2604.23478].

## 5. Mitigation Strategies and Best Practices

### 5.1. Prompt Ensembling and Averaging

Averaging results over multiple semantically equivalent prompts minimizes prompt-specific noise, leading to more robust aggregate JSS [2506.22316, 2604.24074].

### 5.2. Contrastive Decoding

Subtracting assistant model logits at each decoding step attenuates score-range bias, flattening out JSS and yielding higher mean judge-human agreement [2510.18196].

### 5.3. Direct Reporting and Transparency

Papers recommend routine reporting of JSS or its empirical analogues—prompt sensitivity ranges, flip rates, confidence intervals—alongside standard accuracy, and public release of all judge prompt variants and configurations [2604.23478, 2604.24074].

### 5.4. Calibration and Model Selection

JSS should inform judge model choice: a threshold of JSS $\geq 0.8$ is suggested for productionized pipelines expected to encounter cross-team prompt diversity [2604.23478].

### 5.5. Statistical Inference in Multi-Judge Settings

For HJA-type frameworks, estimates of $\gamma_k$ should include valid confidence intervals, with high-leverage diagnostics used to flag systematic disagreement or outlier judges [2605.05073].

## 6. Limitations and Open Challenges

### 6.1. Confounded Task Setups

In factuality and preference tasks, JSS is frequently confounded by template artifacts (polarity, position) rather than reflecting isolated model sensitivity. This limits the interpretability of stability metrics in current benchmark configurations [2604.23478].

### 6.2. Language and Generality

Current JSS metrics and paraphrase sets are primarily English-only and templated; extension to multilingual and less-structured prompts remains a major gap [2604.23478].

### 6.3. Higher-Order and Cross-Judge Effects

Most reported JSS metrics are single-model, single-task. Systematic multi-judge analysis, especially cross-family or hybrid-panel settings as in HJA [2605.05073], is in its early stages.

### 6.4. Stochasticity Beyond T=0

JSS as reported is typically under deterministic (temperature-zero) decoding. Evaluating stability under sampling, real-world API settings, or in the presence of adversarial paraphrases is an open frontier [2604.23478].

## 7. Summary Table: JSS Formulations and Reporting Contexts

| JSS Variant        | Mathematical Definition                                   | Primary Reference        |
|--------------------|----------------------------------------------------------|-------------------------|
| Paraphrase JSS     | $\frac{1}{|P|} \sum_{i=1}^{|P|} 1[j(p_i) = j(p_i') ]$    | [2604.23478]            |
| Range-Span JSS     | $\max_{r} \rho_r - \min_{r} \rho_r$                      | [2510.18196]            |
| Score-Drop JSS     | $(1/N) \sum_{i=1}^N [s_{i,0} - s_{i,t,k}]$               | [2604.05371]            |
| HJA Sensitivity    | $\gamma_k$ in $S_{ki} = \gamma_k \mu_i + [...]$          | [2605.05073]            |

These operationalizations measure related but distinct sensitivities: to prompt wording, score-range, input corruption, and consensus strength, respectively. Each is appropriate in its canonical setting and responds differently to model architecture, prompt design, and downstream aggregation methodology.

Source: https://www.emergentmind.com/topics/judge-sensitivity-score-jss