Subjectivity Grading in Content Evaluation

Updated 6 July 2026

Subjectivity grading is a task that assigns numerical scores to content based on human impressions, capturing nuances such as personalized voice descriptors and continuous scale ratings.
Methodological approaches include absolute ratings, pairwise comparisons, and probabilistic models to represent annotator disagreement and distinguish valid variations from noise.
Applications span multiple domains—including speech, text, and image aesthetics—enabling personalized assessments, improved reliability, and enhanced calibration of evaluation metrics.

Subjectivity grading is the family of tasks in which a system assigns a score or label to content whose evaluation depends on human impressions, interpretive ambiguity, or graded judgment rather than a single objective truth. In speech, the task was introduced as learning a scoring function $f_\theta(x;D)$ for a subjective voice descriptor such as “cute” or “my favorite voice” (Kondo et al., 24 Jun 2025). Related formulations include document-level subjectivity on a continuous $0$–$100$ scale (Gailit et al., 10 Dec 2025), binary objective-versus-subjective classification in multilingual text (Biswas et al., 2024), ambiguity grading through minimal rewrites of narrative claims (Subbiah et al., 1 Apr 2025), and explicit uncertainty modeling for image aesthetics (Jang et al., 2022). Across these settings, the central technical problem is not only prediction accuracy, but also how to represent inter-annotator variation, choose an annotation scale, and separate inherent subjectivity from difficulty or noise (Jin et al., 2018).

1. Core formulations

A canonical formulation appears in speech impression assessment. A subjective voice descriptor (SVD) is any short phrase that captures a listener’s impression of a voice, and the task is: given an input utterance $x$ and a chosen descriptor $D$ , learn a scoring function $f_\theta(x;D)$ that produces a numerical score $\hat s$ reflecting how strongly $x$ embodies $D$ (Kondo et al., 24 Jun 2025). The formulation is explicitly designed to accommodate SVDs personalized to each individual, including descriptors such as “my favorite voice,” even though the reported study used shared, non-personalized descriptors (Kondo et al., 24 Jun 2025).

Other domains instantiate the same general problem with different label spaces. In Estonian document analysis, each text is rated for subjectivity on a continuous sliding scale from $0$ (fully objective) to $0$0 (fully subjective) (Gailit et al., 10 Dec 2025). In multilingual CheckThat! subjectivity detection, the output space is binary, objective versus subjective, for sentence- or paragraph-level text in Arabic, Bulgarian, English, German, Italian, and a multilingual setting (Biswas et al., 2024). In narrative summarization, the question is reframed from “Is this claim faithful?” to “How subjective or ambiguous is this claim?”, operationalized by measuring how much a summary sentence must be rewritten before it becomes unambiguously faithful (Subbiah et al., 1 Apr 2025).

A distinct but closely related line of work treats subjectivity as a latent property of annotator populations rather than only of items. The SDR model for crowdsourcing explicitly encodes question difficulty as a model parameter and implicitly encodes question subjectivity via latent preference factors for crowd-workers; subjectivity is then associated with the number of distinct “true answers” perceived by different worker groups (Jin et al., 2018). This suggests that subjectivity grading is not restricted to end-task scoring, but also concerns the statistical structure of disagreement itself.

2. Annotation paradigms and scale design

Recent work uses several recurrent annotation paradigms: absolute ratings, pairwise or comparative judgments, continuous scales, categorical multidimensional labels, and rewrite-based grading. Their practical differences are consequential because the chosen scale changes both reliability and model behavior (Kondo et al., 24 Jun 2025, Wolfert et al., 2021, Li et al., 6 Jan 2026).

Setting	Label structure	Reported evaluation
Speech impressions	ACR: $0$1–$0$2; CCR: $0$3	ppref (Kondo et al., 24 Jun 2025)
Estonian document subjectivity	Continuous $0$4–$0$5	Pearson’s $0$6 (Gailit et al., 10 Dec 2025)
Narrative faithfulness ambiguity	Rewrite / no rewrite, word-level Levenshtein distance, $0$7	balanced accuracy, annotator agreement (Subbiah et al., 1 Apr 2025)
Earnings-call QA	Six features, each on $0$8	weighted F1 (Pardawala et al., 2024)
LLM-as-a-judge	$0$9–$100$0, $100$1–$100$2, $100$3–$100$4	ICC, nMAE (Li et al., 6 Jan 2026)

In speech, Absolute Category Rating (ACR) asks annotators to listen to a single sample and choose an integer rating $100$5 from “1: not so” to “5: so,” whereas Comparison Category Rating (CCR) asks annotators to compare a pair $100$6 and assign one of four labels: “$100$7 is more so,” “$100$8 is a little more so,” “$100$9 is a little more so,” or “ $x$ 0 is more so,” mapped to $x$ 1 (Kondo et al., 24 Jun 2025). The speech study states that CCR captures a relative preference and is often more reliable and easier for naïve listeners than fine-grained absolute scales (Kondo et al., 24 Jun 2025).

The pairwise-versus-rating distinction also appears in co-speech gesture evaluation. There, pairwise comparisons asked which video was more human-like, whereas rating scales used a $x$ 2– $x$ 3 ordinal scale from “not at all” to “extremely” human-like; both approaches were able to rank the videos according to quality and the ranking significantly correlated, but pairwise comparisons were slightly faster and came with improved inter-rater reliability (Wolfert et al., 2021).

Scale granularity matters even when raters are humans and LLMs rather than only humans. A dedicated study comparing $x$ 4– $x$ 5, $x$ 6– $x$ 7, and $x$ 8– $x$ 9 scales across six benchmarks found that aggregated over tasks, the grading scale of $D$ 0– $D$ 1 yields the strongest human–LLM alignment by absolute-agreement ICC, even though within-group panel reliability remained high on all scales (Li et al., 6 Jan 2026). In that study, fractional values were allowed in all cases to isolate effects of numeric range rather than discretization (Li et al., 6 Jan 2026).

3. Modeling strategies and evaluation metrics

Modeling strategies for subjectivity grading range from direct regression and ranking losses to probabilistic uncertainty models, hybrid feature systems, and LLM-based rewriting. In speech, conventional ACR-based training regresses a scalar score from either a CNN-BLSTM-FC backbone over a $D$ 2-dim magnitude spectrogram or a fixedSSL-FC model over $D$ 3-dim time-averaged wav2vec2.0 features, using mean squared error,

$D$ 4

The CCR alternative keeps the same backbone but uses RankNet learning on annotated pairs with

$D$ 5

and a binary cross-entropy loss over soft labels $D$ 6 (Kondo et al., 24 Jun 2025).

The principal evaluation metric in that setting is precision of preferences, ppref, which measures the fraction of test pairs whose predicted ordering matches the annotated CCR. With $D$ 7 the set of annotated test pairs and $D$ 8 the model scores,

$D$ 9

Two variants are reported: ppref-strong for “more so” labels and ppref-weak for “a little more” labels (Kondo et al., 24 Jun 2025).

A different formalization appears in image aesthetics. There, the observed vote histogram is fit by a Beta distribution with parameters $f_\theta(x;D)$ 0, and subjective logic masses for belief, disbelief, and uncertainty are computed in closed form. The uncertainty mass

$f_\theta(x;D)$ 1

is defined as the subjectivity score, called Aesthetic Uncertainty (AesU) (Jang et al., 2022). The associated prediction model jointly outputs a $f_\theta(x;D)$ 2-dimensional rating histogram and $f_\theta(x;D)$ 3, trained with a composite loss that combines histogram EMD, RMSLE on Beta parameters, and EMD between the predicted histogram and the discretized Beta law (Jang et al., 2022).

Narrative ambiguity grading replaces scalar scoring with minimal editing. For a summary sentence $f_\theta(x;D)$ 4, an LLM-based rewrite model produces a rewritten claim $f_\theta(x;D)$ 5 and a list of explanation points $f_\theta(x;D)$ 6. The Ambiguity Rewrite Metric defines $f_\theta(x;D)$ 7, uses the word-level Levenshtein distance between $f_\theta(x;D)$ 8 and $f_\theta(x;D)$ 9, and counts $\hat s$ 0 as a further continuous signal of how much revision was needed (Subbiah et al., 1 Apr 2025).

In crowdsourcing, the SDR model separates subjectivity from difficulty through a latent-variable generative process. Workers have expertise parameters and latent preference distributions, questions have difficulty parameters, and a Monte Carlo estimate of the expected number of distinct cluster-specific truths yields a question-level subjectivity score $\hat s$ 1 (Jin et al., 2018). This is one of the clearest formal attempts to distinguish “different valid answers” from ordinary labeling error.

Across tasks, evaluation metrics vary with the output type: Pearson’s $\hat s$ 2 for continuous document-level subjectivity (Gailit et al., 10 Dec 2025), Macro F1 for binary subjectivity detection (Biswas et al., 2024, Casanova et al., 2024), weighted F1 for six-dimensional QA attributes (Pardawala et al., 2024), PLCC, SROCC, and MAE for aesthetic subjectivity regression (Jang et al., 2022), ICC for human–LLM absolute agreement across grading scales (Li et al., 6 Jan 2026), and quadratic weighted Cohen’s $\hat s$ 3 for ordinal medical grading under inter-observer variation (Wetstein et al., 2020).

4. Empirical regularities

A recurrent empirical finding is that relative judgments often yield a stronger training or evaluation signal than absolute ratings when the target is subjective. In speech impression assessment, even with as few as $\hat s$ 4 CCR training pairs, ppref-strong reached approximately $\hat s$ 5– $\hat s$ 6 depending on the SVD and model backbone, and across all SVDs and both backbones, CCR-trained models consistently outperformed ACR-trained models on ppref-strong, especially in the low-data regime (Kondo et al., 24 Jun 2025). In co-speech gesture evaluation, both rating scales and pairwise comparisons recovered similar rankings of gesture quality, but pairwise comparisons were approximately $\hat s$ 7 seconds faster per judgment and showed higher inter-rater reliability, with ICC $\hat s$ 8 versus $\hat s$ 9 (Wolfert et al., 2021).

A second regularity is that disagreement is structured rather than random. In the Estonian Subjectivity Dataset, full-set initial pairwise correlations between three annotators were approximately $x$ 0, $x$ 1, and $x$ 2, while re-annotation of the most divergent texts raised the post-hoc agreement between two annotators to $x$ 3 (Gailit et al., 10 Dec 2025). In SubjECTive-QA, agreement was highest for Relevant and Clear and much lower for Assertive, Cautious, Optimistic, and Specific, with full three-way agreement of $x$ 4 for Relevant but only $x$ 5 for Assertive and $x$ 6 for Specific (Pardawala et al., 2024). These results indicate that “subjectivity” is dimension-dependent even within a single dataset.

A third regularity is that subjectivity interacts strongly with evaluation protocol. In narrative summarization, presenting annotators with LLM rewrites rather than original claims increased agreement on faithfulness from $x$ 7 to $x$ 8, a $x$ 9 absolute improvement (Subbiah et al., 1 Apr 2025). In LLM-as-a-judge experiments, pooled human–LLM absolute agreement peaked at ICC $D$ 0 on the $D$ 1– $D$ 2 scale, versus $D$ 3 on $D$ 4– $D$ 5 and $D$ 6 on $D$ 7– $D$ 8; however, pooled reliability masked substantial benchmark heterogeneity, with LLM ICC on subjective benchmarks such as MT-Bench and SummEval much lower than on STS-B and ToxiGen (Li et al., 6 Jan 2026).

A fourth regularity is that explicit modeling of uncertainty or auxiliary signals improves subjectivity prediction. In image aesthetics, simultaneous learning of the rating histogram and Beta parameters improved ConvNeXT-T subjectivity prediction from PLCC approximately $D$ 9 to approximately $0$0, with MAE improving from approximately $0$1 to approximately $0$2 (Jang et al., 2022). In English subjectivity detection, adding semantic embeddings and VAGO scores to RoBERTa increased macro-F1 from $0$3 for the RoBERTa baseline to $0$4 for the final hybrid system on the dev set (Casanova et al., 2024).

5. Domains of use

Speech is the setting in which the label concept itself is most explicit. The speech framework uses $0$5 short Japanese sentences from the APPBLA multi-speaker corpus, read by $0$6 speakers and labeled by $0$7 native Japanese listeners, and it is designed so that a single user’s personal descriptor could be used to collect ACRs and CCRs and fine-tune a model (Kondo et al., 24 Jun 2025). The reported applications are voice search in large corpora, personalized TTS fine-tuning, and voice coaching tools (Kondo et al., 24 Jun 2025).

Text and discourse tasks use subjectivity grading both for direct content analysis and for evaluation of generated outputs. Estonian document-level subjectivity provides a language resource with human and GPT-5 scores on $0$8 texts (Gailit et al., 10 Dec 2025). CheckThat! systems cast subjectivity as multilingual sequence classification and report competitive Macro F1 values such as $0$9 on German and $0$00 on the multilingual set for one system, and $0$01 on English and $0$02 on the multilingual set for another (Biswas et al., 2024, Casanova et al., 2024). In earnings-call QA, subjectivity is decomposed into six features—Assertive, Cautious, Optimistic, Specific, Clear, and Relevant—on a three-point scale, and transfer to White House Press Briefings and Gaggles yields a mean weighted F1 score of $0$03 (Pardawala et al., 2024).

Educational grading treats subjectivity as inconsistency across graders, prompts, or model configurations. In automatic short-answer grading, a transformer regressor evaluated against historic legally binding grades achieved a median absolute error $0$04 smaller than certified human re-graders (Gobrecht et al., 2024). AI-PAT graded over $0$05 exam submissions and handled $0$06 appeal cases; the appeal process led to grade changes in $0$07 of cases, while survey responses showed speed and feedback benefits alongside trust and fairness concerns (Aytutuldu et al., 18 Apr 2025). A separate unified LLM-enhanced framework integrates Textual Similarity Matching, Key Points Matching, Pseudo-Question Generation and Matching, and LLM-Based General Evaluation, and reports deployment in real-world training and certification exams at a major e-commerce enterprise (Zhua et al., 9 Oct 2025).

Medical grading provides another important use case because inter-observer variability is often clinically consequential. In DCIS grading, manual grading shows significant inter-observer variation, and a DenseNet-based system trained on consensus grades achieved lesion-level and patient-level quadratic weighted $0$08 values similar to expert observers (Wetstein et al., 2020). In blastocyst grading, embryo assessment is described as relying on visual inspection of subtle morphological cues, with literature Cohen’s $0$09 values of $0$10–$0$11; the MEmEBG multitask embedding approach improved average F1 on TE and EXP relative to single-task learning (Angabini et al., 14 Apr 2026). A plausible implication is that subjectivity grading methods are especially valuable where labels are ordinal, scarce, and operational decisions depend on consistency.

6. Reliability, controversy, and future directions

A central misconception is that annotator disagreement is equivalent to low-quality annotation. Several lines of work argue against this. In crowdsourcing, question subjectivity and question difficulty are distinct latent causes of response variation (Jin et al., 2018). In narrative summarization, disagreement often comes from wording overlap with multiple interpretations, minor assumed details not stated in the text, skipped causal links, or conclusions made explicit that the story left ambiguous (Subbiah et al., 1 Apr 2025). In speech, the framework is motivated precisely by descriptors that may be personal rather than shared (Kondo et al., 24 Jun 2025). These results support treating disagreement as a first-class signal.

Another recurring issue is whether LLMs can replace human graders or annotators. The Estonian dataset reports that GPT-5 scores were similar to human annotators and comparable to human–human agreement by Pearson’s $0$12, but also that several divergence patterns emerged: GPT-5 treated quoted speech in news as subjective content, humans upweighted colloquial or forum tone, and GPT-5 never output $0$13 while humans used both extremes; the paper therefore states that LLM-based automatic subjectivity scoring is feasible but is not an interchangeable alternative to human annotation (Gailit et al., 10 Dec 2025). In education, strong session-to-session LLM stability coexists with fairness concerns, and the recommended response is transparent grading rubrics, human oversight, and appeal mechanisms (Aytutuldu et al., 18 Apr 2025). In LLM-as-a-judge studies, pooled reliability can create a “reliability illusion,” making per-benchmark and per-subgroup ICC diagnostics essential (Li et al., 6 Jan 2026).

Methodologically, future directions are already explicit in the literature. For speech impressions, the proposed extensions are a richer palette of descriptors, multi-task training across SVDs, integration into TTS and voice-conversion pipelines, and efficient personalized annotation interfaces such as active learning on CCR pairs (Kondo et al., 24 Jun 2025). For document-level subjectivity, recommended practice includes re-annotating the most divergent items, adding a fourth annotator when budget allows, running multiple LLM seeds or batches, and manually inspecting systematic divergence (Gailit et al., 10 Dec 2025). For multilingual subjectivity detection, both translation-based systems note translation noise and suggest language-specific resources or cross-lingual models as alternatives (Biswas et al., 2024, Casanova et al., 2024).

Taken together, the modern subjectivity grading task is not a single benchmark but a methodological pattern: define a subjective target, select an annotation scale that matches the judgment, model disagreement rather than collapsing it away, and evaluate with metrics that reflect the intended use of the scores. Across speech, text, narrative evaluation, aesthetics, education, crowdsourcing, and medicine, the field has moved from treating subjectivity as nuisance variation toward treating it as a measurable property of data, raters, and tasks (Kondo et al., 24 Jun 2025, Jang et al., 2022, Jin et al., 2018).