Score Granularity Gap in AI Evaluation

• Score granularity gap is the mismatch between scoring resolution and the semantically meaningful task granularity, affecting diverse applications like dialogue segmentation and LLM classification.
• Evaluation methods such as boundary F1 and threshold selection often misrepresent performance by conflating distinct granularity levels into a single scalar metric.
• Explicitly controlling granularity through multi-component evaluation and tailored thresholds enhances interpretability and alignment between model outputs and operational needs.

Score granularity gap denotes a mismatch between the granularity at which a score is produced, rewarded, or thresholded and the granularity at which the underlying task is semantically meaningful or operationally useful. In dialogue topic segmentation, it is defined as the gap between actual segmentation quality and what F1-like metrics reward, namely boundary count alignment and exact placement at the annotation’s granularity (Coen, 18 Dec 2025). In black-box LLM classification, it denotes the mismatch between ranking quality and the number of distinct thresholds a confidence score affords for selective prediction (Sun et al., 20 Jun 2026). Closely related formulations appear in open-domain question answering, where coarse-but-correct answers are hidden by single-granularity references, and in acoustic modeling, where scores are computed at frame level while evaluation is at token level (Yona et al., 2024, Xu et al., 2023).

1. Conceptual structure

Across the literature, granularity refers to the level of resolution at which units are represented or judged. In dialogue segmentation, granularity is the density of boundaries and the resolution of segmentation, ranging from coarse major task transitions to fine subtopic or discourse-phase transitions (Coen, 18 Dec 2025). In text analysis, granularity is defined as the level of abstraction at which referential expressions are realized, and is explicitly distinguished from sentence specificity, detail, and verbosity (Ellinger et al., 26 May 2026). In black-box classification, the issue is not semantic abstraction but score resolution: a confidence score may be well ranked and even well calibrated, yet take only a few distinct values, leaving only a few usable thresholds for selective prediction (Sun et al., 20 Jun 2026).

These formulations share a common pattern. A score can collapse distinct phenomena into a single scalar: boundary misplacement and granularity mismatch in segmentation, correctness and informativeness in question answering, or ranking quality and threshold controllability in selective prediction. This suggests a broader family of failures in which a metric or confidence construction is locally plausible but globally misaligned with the level at which decisions are actually made.

A recurrent misconception is that a high-quality score is fully characterized by ranking or calibration. The classification study explicitly separates ranking quality, calibration, and score granularity, and shows that monotone calibration cannot increase granularity because it preserves the effective set of usable thresholds (Sun et al., 20 Jun 2026). The dialogue study makes a parallel distinction between detection failure and granularity mismatch: low F1 can arise either from semantically misplaced boundaries or from coherent segmentation at a different scale than the gold annotation (Coen, 18 Dec 2025).

2. Dialogue topic segmentation and the failure of boundary F1

Dialogue topic segmentation provides the clearest formal diagnosis of a score granularity gap. Historically, the task is evaluated as boundary detection: given dialogue turns $m_1,\dots,m_T$, predict a boundary set $B \subseteq \{1,\dots,T-1\}$, and score it with precision, recall, and F1. Because turn-level annotation is noisy, many benchmarks use window-tolerant F1, where a predicted boundary counts as correct if it falls within a fixed window, here $\pm 1$ message, of a gold boundary. The paper argues that all such F1-style metrics implicitly assume a single correct segmentation granularity and treat any deviation in the number of boundaries as error, even when the resulting segments are coherent (Coen, 18 Dec 2025).

The central diagnostic variable is the boundary oversegmentation ratio,

$$\mathrm{BOR} = \frac{\#B_{\mathrm{pred}}}{\#B_{\mathrm{gold}}},$$

which measures how dense the predicted segmentation is relative to the annotation. The proposed evaluation objective reports window-tolerant F1 together with BOR and segment coherence diagnostics, namely purity and coverage. Purity measures the internal coherence of predicted segments with respect to gold topics; coverage measures how well each gold segment is captured by a contiguous predicted segment. The resulting taxonomy distinguishes calibrated segmentation, oversegmentation, undersegmentation, and genuine detection failure.

The score granularity gap appears when F1 or W-F1 is low, BOR is much greater than 1, and purity is high. In that regime, the metric reports poor performance, but the segmentation is simply finer than the gold annotation. Across eight datasets—SuperDialseg, DialSeg711, TIAGE, MultiWOZ, DailyDialog, Taskmaster, Topical-Chat, and QMSum—the authors report high segment coherence combined with extreme oversegmentation relative to sparse labels. A canonical example is DialSeg711 in the Stage 3 final test: $W$-F1 $=0.767$, F1 $=0.434$, BOR $=2.53$, Purity $=0.962$, Coverage $=0.651$, with 5167 predicted versus 2042 gold boundaries. The predicted segments are highly coherent, yet most extra fine-grained boundaries are counted as false positives (Coen, 18 Dec 2025).

The study further shows that density alignment dominates F1. Random or periodic baselines with $B \subseteq \{1,\dots,T-1\}$0 achieve surprisingly high W-F1, and leaderboard gains on SuperDialseg are largely driven by shifts to more aggressive BOR regimes rather than better semantic boundary placement. To address this, the paper separates boundary scoring from boundary selection. A window-vs-window DistilBERT classifier assigns scores $B \subseteq \{1,\dots,T-1\}$1 to candidate positions, and a separate selection rule controls density via a threshold $B \subseteq \{1,\dots,T-1\}$2 and minimum spacing $B \subseteq \{1,\dots,T-1\}$3. This makes granularity an explicit design parameter rather than an accidental byproduct of threshold tuning (Coen, 18 Dec 2025).

3. Score resolution in black-box LLM classification

The black-box LLM classification study gives the most explicit operational definition of score granularity gap. In binary selective prediction, an operator thresholds a scalar score $B \subseteq \{1,\dots,T-1\}$4 to choose which cases are auto-handled and which are routed to human review. The paper asks not only whether the score ranks examples well, but at what resolution it can be thresholded. Its basic resolution metric is

$B \subseteq \{1,\dots,T-1\}$5

the number of distinct score values observed on the test set. Two supplementary measures quantify spread and usable interior mass: $B \subseteq \{1,\dots,T-1\}$6 Here $B \subseteq \{1,\dots,T-1\}$7 is score entropy over 20 equal-width bins, and $B \subseteq \{1,\dots,T-1\}$8 is the fraction of scores away from the extremes (Sun et al., 20 Jun 2026).

The study compares seven ways to build confidence scores across 25 model-dataset pairs involving 9 LLMs and 3 benchmarks. A central result is that single-shot verbalized confidence, once converted to class probability, ranks cases surprisingly well but is extremely coarse. Its normalized PR-AUC ranges from 0.41 to 0.95, with mean 0.80, yet it takes only 1–21 distinct values, mean 9. On average, 45% of test examples share its most common score value. In a representative MNLI example with GPT-4o-mini, Verb yields PR-AUC $B \subseteq \{1,\dots,T-1\}$9 but only 4 distinct values, so the risk–coverage curve becomes a four-step staircase (Sun et al., 20 Jun 2026).

Multi-query aggregation widens resolution dramatically. Logistic regression over sub-task features or over multi-prompt paraphrase features raises score granularity to a mean of 138 distinct values and raises score entropy to about 2.8 nats, while substantially reducing collocation. Calibrated sub-task aggregation also lifts interior mass to roughly 45%. Yet the ranking gains are conditional. Multi-query aggregation helps weaker models, but can degrade already-strong ones; the paper reports a negative correlation between the PR-AUC gain from MP-LR over Verb and the model’s baseline accuracy, $\pm 1$0, $\pm 1$1. The practical conclusion is that ranking quality, calibration, and threshold resolution are separate deployment properties, and that calibration alone cannot manufacture granularity that the base score does not contain (Sun et al., 20 Jun 2026).

4. Multi-granularity answers and granularity-sensitive text scoring

Open-domain question answering exposes the same phenomenon in evaluative form. GRANOLA QA replaces a single answer set with an ordered sequence of answer sets by granularity,

$\pm 1$2

where $\pm 1$3 contains the most informative correct answers and later sets become progressively coarser. If a prediction first matches level $\pm 1$4, GRANOLA Accuracy is

$\pm 1$5

and GRANOLA Informativeness is

$\pm 1$6

This explicitly separates correctness from informativeness. On GRANOLA-EQ, standard decoding often produces specific answers that are incorrect, whereas Decoding with Response Aggregation (DRAG) aligns answer granularity with uncertainty and yields a nearly 20 point increase in accuracy on average when evaluated on multi-granularity answers, with larger gains for rare entities (Yona et al., 2024).

Granuscore addresses a related problem without references. It defines granularity as the level of abstraction at which referential expressions are realized, embeds concepts in a hierarchical hyperbolic space, and maps them to a percentile score where lower values are finer and higher values are coarser. In question answering, it reveals a measurable granularity mismatch between model answers and gold answers: $\pm 1$7 A logistic regression using only $\pm 1$8 predicts correctness with AUC $\pm 1$9. At the dataset level, FACTS Parametric has the lowest gold-answer granularity and the lowest mean accuracy, about 5%, whereas TruthfulQA has the highest gold-answer granularity and the highest mean accuracy, about 43%. The paper’s broader claim is that conventional QA metrics have no notion of whether an answer is at the right level of abstraction, so they conflate factual failure with wrong-level content (Ellinger et al., 26 May 2026).

Taken together, these QA results recast the gap as a mismatch between what is known and what a benchmark can recognize. A coarse-but-correct answer such as “1961” may be invisible under single-granularity evaluation, just as a coherent fine-grained segmentation can be invisible under strict boundary F1.

5. Cross-domain manifestations

Closely related score granularity gaps appear in scientific retrieval. MixGR starts from the observation that scientific queries often contain multiple subqueries and that evidence is distributed across propositions inside long documents. It defines three scores—query–document, query–proposition, and subquery–proposition—and fuses their rankings with Reciprocal Rank Fusion: $$\mathrm{BOR} = \frac{\#B_{\mathrm{pred}}}{\#B_{\mathrm{gold}}},$$0 On queries containing multiple subqueries, MixGR improves nDCG@5 by 24.7% on average for unsupervised retrievers and 9.8% for supervised retrievers, indicating that a single coarse query–document score misses fine-grained evidence alignment (Cai et al., 2024).

Ontology-grounded skill decomposition exhibits a structural version of the same problem. The evaluation framework aligns generated skill phrases to ESCO descendants and computes both a semantic F1 and a hierarchy-aware F1. The hierarchy-aware score assigns full credit to exact children, half credit to deeper descendants, and zero otherwise: $$\mathrm{BOR} = \frac{\#B_{\mathrm{pred}}}{\#B_{\mathrm{gold}}},$$1 Across models, semantic F1 is typically in the 0.39–0.49 range, while Hier-F1 is much lower, typically 0.04–0.14. The gap between these metrics exposes semantically plausible but structurally mis-granular predictions (Luyen et al., 13 Oct 2025).

In acoustic modeling, the phrase “granularity gap” refers to the mismatch between fine-grained frame-level features and coarser text units. LibriSpeech examples often have a frame-to-subword ratio of 20–50. Progressive Down-Sampling (PDS) addresses this by alternating context interaction and down-sampling so that encoder states become more text-like, reducing the mismatch between where attention scores are computed and where evaluation occurs. With representation fusion, PDS compresses acoustic sequences to $$\mathrm{BOR} = \frac{\#B_{\mathrm{pred}}}{\#B_{\mathrm{gold}}},$$2 of the original length and yields inference speedups from $$\mathrm{BOR} = \frac{\#B_{\mathrm{pred}}}{\#B_{\mathrm{gold}}},$$3 to $$\mathrm{BOR} = \frac{\#B_{\mathrm{pred}}}{\#B_{\mathrm{gold}}},$$4, while attention becomes more concentrated and better aligned with token-level decisions (Xu et al., 2023).

Document layout analysis shows a metric-design counterpart. The COTe framework replaces generic IoU/F1/mAP evaluation with Coverage, Overlap, Trespass, and Excess, computed over Structural Semantic Units. Its main score is

$$\mathrm{BOR} = \frac{\#B_{\mathrm{pred}}}{\#B_{\mathrm{gold}}},$$5

The paper reports that COTe reduces the interpretation-performance gap by up to 76% relative to F1 and is robust to annotation granularity even without explicit SSU labeling. This is another instance in which a single conventional score is too coarse to reflect task-relevant structure (Bourne et al., 13 Mar 2026).

Vision segmentation and user-facing uncertainty displays illustrate control-side manifestations. UnSAMv2 augments SAM-2 with a continuous granularity scalar $$\mathrm{BOR} = \frac{\#B_{\mathrm{pred}}}{\#B_{\mathrm{gold}}},$$6, encoded via Fourier features and an MLP, so that a point prompt and a granularity value jointly determine the mask scale. With only 6K unlabeled images and less than 0.02% additional parameters, it improves $$\mathrm{BOR} = \frac{\#B_{\mathrm{pred}}}{\#B_{\mathrm{gold}}},$$7 from 5.69 to 4.75, 1-IoU from 58.0 to 73.1, and $$\mathrm{BOR} = \frac{\#B_{\mathrm{pred}}}{\#B_{\mathrm{gold}}},$$8 from 49.6 to 68.3 (Yu et al., 17 Nov 2025). In a different direction, uncertainty communication for medical QA shows that the level at which uncertainty is expressed changes behavior: token-level uncertainty increased agreement with the AI, whereas output- and relation-level uncertainty reduced users’ confidence in their own answers, and relation-level uncertainty also reduced external verification such as internet searches and checking provided URLs (Villavicencio et al., 27 May 2026).

6. Methodological consequences, limitations, and open directions

A consistent methodological lesson is that single-number evaluation is often too coarse. Dialogue segmentation recommends a minimum reporting standard consisting of a boundary accuracy metric such as W-F1, boundary density relative to gold such as BOR, and at least one segment-level coherence diagnostic such as purity or coverage (Coen, 18 Dec 2025). GRANOLA QA similarly separates correctness from informativeness rather than collapsing both into lexical overlap (Yona et al., 2024). In tabular probabilistic prediction, ScoringBench makes the same point in a different form: RMSE, MAE, and $$\mathrm{BOR} = \frac{\#B_{\mathrm{pred}}}{\#B_{\mathrm{gold}}},$$9 are too coarse for models that output full predictive distributions, so evaluation should include proper scoring rules such as CRPS, CRLS, Interval Score, weighted CRPS, log score, and Brier score (Landsgesell et al., 31 Mar 2026).

A second recurring theme is that granularity should be made explicit and controllable. Dialogue segmentation operationalizes this by separating boundary scoring from boundary selection and tuning threshold $W$0, spacing $W$1, or target selection rate (Coen, 18 Dec 2025). UnSAMv2 makes mask scale a continuous conditioning variable (Yu et al., 17 Nov 2025). The Granularity Axis work suggests a related latent-control view: in role-conditioned LLMs, a contrast-defined micro-to-macro axis aligns with PC1 at cosine 0.972 in Qwen3-8B, explains 52.6% of variance, and steering along this axis moves Llama-3.1-8B-Instruct from 2.00 to 3.17 on a five-point macro scale for prompts that admit local responses (Qin et al., 7 May 2026). This suggests that some score granularity gaps are not merely evaluative artifacts but also controllable properties of internal representations.

The literature also converges on several limitations. Many diagnostics remain relative to a fixed gold scheme. Dialogue purity and coverage are still measured with respect to gold annotation whose granularity may itself be mismatched (Coen, 18 Dec 2025). Ontology-grounded skill evaluation depends on ontology coverage and threshold choices, so true sub-skills outside the ontology can be marked unverifiable (Luyen et al., 13 Oct 2025). Granuscore is reference-free, but its hierarchical structure is WordNet-based and context-agnostic, so domains underrepresented there may receive noisier scores (Ellinger et al., 26 May 2026). In black-box classification, the most attractive low-cost alternative to coarse verbalized confidence—token log-probabilities with temperature scaling—was only available for 2 of 9 models because most APIs did not expose token log-probs (Sun et al., 20 Jun 2026).

The broader implication is not that one universal replacement metric exists. Rather, the shared lesson is that granularity must be treated as a first-class property of both scoring and control. Where a task supports multiple valid abstraction levels, score design that ignores those levels can mistake coherent fine detail for error, hide partial knowledge, or leave operators with only a few coarse decision thresholds. Where granularity is made explicit—through multi-component evaluation, hierarchical alignment, continuous control variables, or richer probabilistic scoring—the gap between measured performance and task-relevant performance becomes smaller, and the behavior of the system becomes more interpretable and more controllable.