---
title: Subjectivity Grading in Content Evaluation
url: https://www.emergentmind.com/topics/subjectivity-grading-task
type: topic
---

# Subjectivity Grading in Content Evaluation

Subjectivity grading is the family of tasks in which a system assigns a score or label to content whose evaluation depends on human impressions, interpretive ambiguity, or graded judgment rather than a single objective truth. In speech, the task was introduced as learning a scoring function $f_\theta(x;D)$ for a subjective voice descriptor such as “cute” or “my favorite voice” [2506.19335]. Related formulations include document-level subjectivity on a continuous $0$–$100$ scale [2512.09634], binary objective-versus-subjective classification in multilingual text [2407.10252], ambiguity grading through minimal rewrites of narrative claims [2504.01132], and explicit uncertainty modeling for image aesthetics [2208.09666]. Across these settings, the central technical problem is not only prediction accuracy, but also how to represent inter-annotator variation, choose an annotation scale, and separate inherent subjectivity from difficulty or noise [1802.04009].

## 1. Core formulations

A canonical formulation appears in speech impression assessment. A subjective voice descriptor (SVD) is any short phrase that captures a listener’s impression of a voice, and the task is: given an input utterance $x$ and a chosen descriptor $D$, learn a scoring function $f_\theta(x;D)$ that produces a numerical score $\hat s$ reflecting how strongly $x$ embodies $D$ [2506.19335]. The formulation is explicitly designed to accommodate SVDs personalized to each individual, including descriptors such as “my favorite voice,” even though the reported study used shared, non-personalized descriptors [2506.19335].

Other domains instantiate the same general problem with different label spaces. In Estonian document analysis, each text is rated for subjectivity on a continuous sliding scale from $0$ (fully objective) to $100$ (fully subjective) [2512.09634]. In multilingual CheckThat! subjectivity detection, the output space is binary, objective versus subjective, for sentence- or paragraph-level text in Arabic, Bulgarian, English, German, Italian, and a multilingual setting [2407.10252]. In narrative summarization, the question is reframed from “Is this claim faithful?” to “How subjective or ambiguous is this claim?”, operationalized by measuring how much a summary sentence must be rewritten before it becomes unambiguously faithful [2504.01132].

A distinct but closely related line of work treats subjectivity as a latent property of annotator populations rather than only of items. The SDR model for crowdsourcing explicitly encodes question difficulty as a model parameter and implicitly encodes question subjectivity via latent preference factors for crowd-workers; subjectivity is then associated with the number of distinct “true answers” perceived by different worker groups [1802.04009]. This suggests that subjectivity grading is not restricted to end-task scoring, but also concerns the statistical structure of disagreement itself.

## 2. Annotation paradigms and scale design

Recent work uses several recurrent annotation paradigms: absolute ratings, pairwise or comparative judgments, continuous scales, categorical multidimensional labels, and rewrite-based grading. Their practical differences are consequential because the chosen scale changes both reliability and model behavior [2506.19335][2108.05709][2601.03444].

| Setting | Label structure | Reported evaluation |
|---|---|---|
| Speech impressions | ACR: $1$–$5$; CCR: $\{0,0.25,0.75,1\}$ | ppref [2506.19335] |
| Estonian document subjectivity | Continuous $0$–$100$ | Pearson’s $r$ [2512.09634] |
| Narrative faithfulness ambiguity | Rewrite / no rewrite, word-level Levenshtein distance, $|E|$ | balanced accuracy, annotator agreement [2504.01132] |
| Earnings-call QA | Six features, each on $0/1/2$ | weighted F1 [2410.20651] |
| LLM-as-a-judge | $0$–$5$, $0$–$10$, $0$–$100$ | ICC, nMAE [2601.03444] |

In speech, Absolute Category Rating (ACR) asks annotators to listen to a single sample and choose an integer rating $m_i\in\{1,\dots,5\}$ from “1: not so” to “5: so,” whereas Comparison Category Rating (CCR) asks annotators to compare a pair $(i,j)$ and assign one of four labels: “$i$ is more so,” “$i$ is a little more so,” “$j$ is a little more so,” or “$j$ is more so,” mapped to $P_{ij}\in\{0,0.25,0.75,1\}$ [2506.19335]. The speech study states that CCR captures a relative preference and is often more reliable and easier for naïve listeners than fine-grained absolute scales [2506.19335].

The pairwise-versus-rating distinction also appears in co-speech gesture evaluation. There, pairwise comparisons asked which video was more human-like, whereas rating scales used a $1$–$5$ ordinal scale from “not at all” to “extremely” human-like; both approaches were able to rank the videos according to quality and the ranking significantly correlated, but pairwise comparisons were slightly faster and came with improved inter-rater reliability [2108.05709].

Scale granularity matters even when raters are humans and LLMs rather than only humans. A dedicated study comparing $0$–$5$, $0$–$10$, and $0$–$100$ scales across six benchmarks found that aggregated over tasks, the grading scale of $0$–$5$ yields the strongest human–LLM alignment by absolute-agreement ICC, even though within-group panel reliability remained high on all scales [2601.03444]. In that study, fractional values were allowed in all cases to isolate effects of numeric range rather than discretization [2601.03444].

## 3. Modeling strategies and evaluation metrics

Modeling strategies for subjectivity grading range from direct regression and ranking losses to probabilistic uncertainty models, hybrid feature systems, and LLM-based rewriting. In speech, conventional ACR-based training regresses a scalar score from either a CNN-BLSTM-FC backbone over a $257$-dim magnitude spectrogram or a fixedSSL-FC model over $768$-dim time-averaged wav2vec2.0 features, using mean squared error,
$$
L_{\mathrm{ACR}}=\frac{1}{N}\sum_i(\hat s_i-m_i)^2.
$$
The CCR alternative keeps the same backbone but uses RankNet learning on annotated pairs with
$$
\hat P_{i,j}=\sigma(\hat s_j-\hat s_i),\qquad \sigma(u)=\frac{1}{1+e^{-u}},
$$
and a binary cross-entropy loss over soft labels $P_{i,j}\in\{0,0.25,0.75,1\}$ [2506.19335].

The principal evaluation metric in that setting is precision of preferences, ppref, which measures the fraction of test pairs whose predicted ordering matches the annotated CCR. With $D_{\mathrm{CCR}}$ the set of annotated test pairs and $\hat s_i,\hat s_j$ the model scores,
$$
p_{\mathrm{pref}}=\frac{1}{|D_{\mathrm{CCR}}|}\sum_{(i,j)\in D_{\mathrm{CCR}}} I[\hat s_i>\hat s_j].
$$
Two variants are reported: ppref-strong for “more so” labels and ppref-weak for “a little more” labels [2506.19335].

A different formalization appears in image aesthetics. There, the observed vote histogram is fit by a Beta distribution with parameters $(\alpha,\beta)$, and subjective logic masses for belief, disbelief, and uncertainty are computed in closed form. The uncertainty mass
$$
u=\frac{2}{\alpha+\beta}
$$
is defined as the subjectivity score, called Aesthetic Uncertainty (AesU) [2208.09666]. The associated prediction model jointly outputs a $10$-dimensional rating histogram and $(\hat\alpha,\hat\beta)$, trained with a composite loss that combines histogram EMD, RMSLE on Beta parameters, and EMD between the predicted histogram and the discretized Beta law [2208.09666].

Narrative ambiguity grading replaces scalar scoring with minimal editing. For a summary sentence $s$, an LLM-based rewrite model produces a rewritten claim $r$ and a list of explanation points $E$. The Ambiguity Rewrite Metric defines $\mathrm{ARM}_{\mathrm{binary}}(s)=1[r\neq s]$, uses the word-level Levenshtein distance between $s$ and $r$, and counts $|E|$ as a further continuous signal of how much revision was needed [2504.01132].

In crowdsourcing, the SDR model separates subjectivity from difficulty through a latent-variable generative process. Workers have expertise parameters and latent preference distributions, questions have difficulty parameters, and a Monte Carlo estimate of the expected number of distinct cluster-specific truths yields a question-level subjectivity score $S_j$ [1802.04009]. This is one of the clearest formal attempts to distinguish “different valid answers” from ordinary labeling error.

Across tasks, evaluation metrics vary with the output type: Pearson’s $r$ for continuous document-level subjectivity [2512.09634], Macro F1 for binary subjectivity detection [2407.10252][2407.03770], weighted F1 for six-dimensional QA attributes [2410.20651], PLCC, SROCC, and MAE for aesthetic subjectivity regression [2208.09666], ICC for human–LLM absolute agreement across grading scales [2601.03444], and quadratic weighted Cohen’s $\kappa$ for ordinal medical grading under inter-observer variation [2010.03244].

## 4. Empirical regularities

A recurrent empirical finding is that relative judgments often yield a stronger training or evaluation signal than absolute ratings when the target is subjective. In speech impression assessment, even with as few as $125$ CCR training pairs, ppref-strong reached approximately $0.65$–$0.75$ depending on the SVD and model backbone, and across all SVDs and both backbones, CCR-trained models consistently outperformed ACR-trained models on ppref-strong, especially in the low-data regime [2506.19335]. In co-speech gesture evaluation, both rating scales and pairwise comparisons recovered similar rankings of gesture quality, but pairwise comparisons were approximately $6$ seconds faster per judgment and showed higher inter-rater reliability, with ICC $0.77$ versus $0.62$ [2108.05709].

A second regularity is that disagreement is structured rather than random. In the Estonian Subjectivity Dataset, full-set initial pairwise correlations between three annotators were approximately $0.570$, $0.525$, and $0.627$, while re-annotation of the most divergent texts raised the post-hoc agreement between two annotators to $0.678$ [2512.09634]. In SubjECTive-QA, agreement was highest for Relevant and Clear and much lower for Assertive, Cautious, Optimistic, and Specific, with full three-way agreement of $72.6\%$ for Relevant but only $35.1\%$ for Assertive and $38.8\%$ for Specific [2410.20651]. These results indicate that “subjectivity” is dimension-dependent even within a single dataset.

A third regularity is that subjectivity interacts strongly with evaluation protocol. In narrative summarization, presenting annotators with LLM rewrites rather than original claims increased agreement on faithfulness from $36.4\%$ to $57.5\%$, a $21.1\%$ absolute improvement [2504.01132]. In LLM-as-a-judge experiments, pooled human–LLM absolute agreement peaked at ICC$(A,1)=0.853$ on the $0$–$5$ scale, versus $0.805$ on $0$–$10$ and $0.840$ on $0$–$100$; however, pooled reliability masked substantial benchmark heterogeneity, with LLM ICC on subjective benchmarks such as MT-Bench and SummEval much lower than on STS-B and ToxiGen [2601.03444].

A fourth regularity is that explicit modeling of uncertainty or auxiliary signals improves subjectivity prediction. In image aesthetics, simultaneous learning of the rating histogram and Beta parameters improved ConvNeXT-T subjectivity prediction from PLCC approximately $0.2750$ to approximately $0.3331$, with MAE improving from approximately $0.0343$ to approximately $0.0333$ [2208.09666]. In English subjectivity detection, adding semantic embeddings and VAGO scores to RoBERTa increased macro-F1 from $0.7860$ for the RoBERTa baseline to $0.8173$ for the final hybrid system on the dev set [2407.03770].

## 5. Domains of use

Speech is the setting in which the label concept itself is most explicit. The speech framework uses $15$ short Japanese sentences from the APPBLA multi-speaker corpus, read by $2{,}137$ speakers and labeled by $50$ native Japanese listeners, and it is designed so that a single user’s personal descriptor could be used to collect ACRs and CCRs and fine-tune a model [2506.19335]. The reported applications are voice search in large corpora, personalized TTS fine-tuning, and voice coaching tools [2506.19335].

Text and discourse tasks use subjectivity grading both for direct content analysis and for evaluation of generated outputs. Estonian document-level subjectivity provides a language resource with human and GPT-5 scores on $1{,}000$ texts [2512.09634]. CheckThat! systems cast subjectivity as multilingual sequence classification and report competitive Macro F1 values such as $0.7908$ on German and $0.7121$ on the multilingual set for one system, and $0.7442$ on English and $0.6849$ on the multilingual set for another [2407.10252][2407.03770]. In earnings-call QA, subjectivity is decomposed into six features—Assertive, Cautious, Optimistic, Specific, Clear, and Relevant—on a three-point scale, and transfer to White House Press Briefings and Gaggles yields a mean weighted F1 score of $65.97\%$ [2410.20651].

Educational grading treats subjectivity as inconsistency across graders, prompts, or model configurations. In automatic short-answer grading, a transformer regressor evaluated against historic legally binding grades achieved a median absolute error $44\%$ smaller than certified human re-graders [2405.04323]. AI-PAT graded over $850$ exam submissions and handled $185$ appeal cases; the appeal process led to grade changes in $74\%$ of cases, while survey responses showed speed and feedback benefits alongside trust and fairness concerns [2504.13557]. A separate unified LLM-enhanced framework integrates Textual Similarity Matching, Key Points Matching, Pseudo-Question Generation and Matching, and LLM-Based General Evaluation, and reports deployment in real-world training and certification exams at a major e-commerce enterprise [2510.07912].

Medical grading provides another important use case because inter-observer variability is often clinically consequential. In DCIS grading, manual grading shows significant inter-observer variation, and a DenseNet-based system trained on consensus grades achieved lesion-level and patient-level quadratic weighted $\kappa$ values similar to expert observers [2010.03244]. In blastocyst grading, embryo assessment is described as relying on visual inspection of subtle morphological cues, with literature Cohen’s $\kappa$ values of $0.3$–$0.6$; the MEmEBG multitask embedding approach improved average F1 on TE and EXP relative to single-task learning [2604.13217]. A plausible implication is that subjectivity grading methods are especially valuable where labels are ordinal, scarce, and operational decisions depend on consistency.

## 6. Reliability, controversy, and future directions

A central misconception is that annotator disagreement is equivalent to low-quality annotation. Several lines of work argue against this. In crowdsourcing, question subjectivity and question difficulty are distinct latent causes of response variation [1802.04009]. In narrative summarization, disagreement often comes from wording overlap with multiple interpretations, minor assumed details not stated in the text, skipped causal links, or conclusions made explicit that the story left ambiguous [2504.01132]. In speech, the framework is motivated precisely by descriptors that may be personal rather than shared [2506.19335]. These results support treating disagreement as a first-class signal.

Another recurring issue is whether LLMs can replace human graders or annotators. The Estonian dataset reports that GPT-5 scores were similar to human annotators and comparable to human–human agreement by Pearson’s $r$, but also that several divergence patterns emerged: GPT-5 treated quoted speech in news as subjective content, humans upweighted colloquial or forum tone, and GPT-5 never output $100$ while humans used both extremes; the paper therefore states that LLM-based automatic subjectivity scoring is feasible but is not an interchangeable alternative to human annotation [2512.09634]. In education, strong session-to-session LLM stability coexists with fairness concerns, and the recommended response is transparent grading rubrics, human oversight, and appeal mechanisms [2504.13557]. In LLM-as-a-judge studies, pooled reliability can create a “reliability illusion,” making per-benchmark and per-subgroup ICC diagnostics essential [2601.03444].

Methodologically, future directions are already explicit in the literature. For speech impressions, the proposed extensions are a richer palette of descriptors, multi-task training across SVDs, integration into TTS and voice-conversion pipelines, and efficient personalized annotation interfaces such as active learning on CCR pairs [2506.19335]. For document-level subjectivity, recommended practice includes re-annotating the most divergent items, adding a fourth annotator when budget allows, running multiple LLM seeds or batches, and manually inspecting systematic divergence [2512.09634]. For multilingual subjectivity detection, both translation-based systems note translation noise and suggest language-specific resources or cross-lingual models as alternatives [2407.10252][2407.03770].

Taken together, the modern subjectivity grading task is not a single benchmark but a methodological pattern: define a subjective target, select an annotation scale that matches the judgment, model disagreement rather than collapsing it away, and evaluate with metrics that reflect the intended use of the scores. Across speech, text, narrative evaluation, aesthetics, education, crowdsourcing, and medicine, the field has moved from treating subjectivity as nuisance variation toward treating it as a measurable property of data, raters, and tasks [2506.19335][2208.09666][1802.04009].

Source: https://www.emergentmind.com/topics/subjectivity-grading-task