OpinionQA: Subjective Question Answering

• OpinionQA is a family of question-answering tasks that focuses on subjective, perspective-dependent answers derived from reviews and surveys.
• It challenges traditional QA by requiring models to handle ambiguity, diverging viewpoints, and demographic-conditioned response distributions.
• Evaluation involves metrics like accuracy and distributional alignment, with techniques incorporating subjectivity supervision and pluralistic aggregation.

OpinionQA denotes a family of question-answering problems in which the target is not a single objectively correct fact but an opinion, experience, judgment, or demographic-conditioned response distribution. In the review-grounded literature, the evidence comes from customer reviews and community answers, and the central difficulties are ambiguity, subjectivity, and diverging viewpoints; in later survey-grounded work, OpinionQA becomes a benchmark for predicting how a specified demographic group answers a multiple-choice question or how its full answer distribution is expressed (Wan et al., 2016, Meister et al., 2024). Across these formulations, the common requirement is to model answers that are conditional on perspective rather than uniquely determined by an external fact.

1. Conceptual foundations

The review-grounded formulation of OpinionQA emerged from the observation that many real queries are not factoid queries at all. Product questions such as whether an item is “loud enough,” “vegan friendly,” or suitable for a particular use case are grounded in user experiences and private states rather than encyclopedic facts (McAuley et al., 2015, Bjerva et al., 2020). In this setting, the standard QA assumption of a single correct answer becomes unstable.

A canonical statement of the problem is that opinion question answering differs from traditional QA in at least two ways: questions may have “multiple, often divergent, answers,” and what counts as a good answer can depend on the asker and the answerer (Wan et al., 2016). SubjQA makes the same point in a different vocabulary, defining subjectivity as the expression of internal opinions or beliefs “which cannot be objectively observed or verified,” and showing that subjectivity matters in QA with “more intricate interactions” than in sentiment classification (Bjerva et al., 2020).

A frequent misconception is that OpinionQA is merely domain transfer from Wikipedia-style reading comprehension to reviews. SubjQA explicitly rejects that interpretation: a subjective question may or may not have a subjective answer, and a factual question can occur in a review context saturated with subjective language (Bjerva et al., 2020). This suggests that OpinionQA is not just QA over a different corpus; it is QA under perspective dependence, mixed evidence, and answer pluralism.

2. Review-grounded corpora and task formulations

An early large-scale formulation appears in “Modeling Ambiguity, Subjectivity, and Diverging Viewpoints in Opinion Question Answering Systems” (Wan et al., 2016). That work argued that review QA must explicitly represent ambiguity and personalization, and reported a dataset with 800 thousand questions—and over 3.1 million answers (Wan et al., 2016).

Representative resources span retrieval-oriented, extractive, projective, generative, and survey-grounded settings.

Resource	Evidence and scale	Main formulation
“Moqa” (McAuley et al., 2015)	1,447,173 questions with answers and 13,498,681 customer reviews	Reviews as experts for binary and open-ended product queries
“ReviewQA” (Grail et al., 2018)	587,492 questions over 100,000 hotel reviews	Projective aspect-based QA with 8 task groups
“SubjQA” (Bjerva et al., 2020)	over 10,000 English examples across 6 distinct domains	Extractive QA with subjectivity labels for questions and answer spans
“OpinionQA” (Zhang et al., 5 Oct 2025)	1,498 multiple-choice survey questions and 91K unique <question, demographic> pairs	Demographic-conditioned majority-answer prediction

The review-grounded benchmarks emphasize different structural properties. ReviewQA organizes hotel-review QA around aspect detection, aspect-specific satisfaction, threshold comparison, exact rating prediction, aspect comparison, and best/worst aspect selection; its answer space is projective rather than extractive (Grail et al., 2018). SubjQA instead focuses on extractive review comprehension with explicit subjectivity labels, span supervision, and realistic unanswerability (Bjerva et al., 2020).

SubjQA is particularly informative about the statistical structure of opinion-oriented QA. It reports 73% subjective questions and 74% subjective answers, with roughly 65% of question-review pairs answerable overall (Bjerva et al., 2020). Its reasoning analysis of 100 sampled answerable development questions assigns 18% to lexical reasoning, 28% to paraphrase, 43% to indirect reasoning, and 11% to noisy or non-applicable cases (Bjerva et al., 2020). Those numbers formalize a core property of OpinionQA: much of the signal is indirect rather than lexically aligned.

3. Modeling architectures

A foundational model for review-based OpinionQA is Moqa, a mixture-of-experts framework in which each review is treated as an expert (McAuley et al., 2015). Its generic prediction rule is

$p(y \mid X) = \sum_f p(f \mid X)\, p(y \mid f, X),$

where the expert is a review, the gate is a learned relevance function, and the expert vote is a learned answer-support function (McAuley et al., 2015). Moqa is designed for heterogeneous linguistic mappings between question language and review language, using BM25+, ROUGE-L, and a bilinear compatibility term in its relevance model (McAuley et al., 2015). On binary yes/no questions, it reports 76.8% average accuracy@50% across the eight largest categories; on open-ended questions, it reports 0.883 average AUC (McAuley et al., 2015).

SubjQA introduces a different modeling question: whether explicit subjectivity supervision improves extractive QA. Its multitask subjectivity-aware model extends FastQA with a shared encoder and a subjectivity classification head trained jointly with span prediction (Bjerva et al., 2020). The paper reports that out-of-the-box SQuAD-pretrained models reach only 30.5% F1 on average across SubjQA domains, while fine-tuned BERT reaches 74.1% F1, and the multitask subjectivity-aware model reaches 76.3% F1 (Bjerva et al., 2020). The broader implication is that subjectivity is not merely metadata; it is useful supervision for answer extraction itself.

ReviewQA, by contrast, is designed to test reasoning over latent aspect structure rather than span extraction. Its strongest baseline, a deep projective reader, reaches 60.4 overall accuracy, with especially strong results on overall satisfaction (90.9), aspect sentiment (85.9), and threshold comparison (91.3), but much weaker performance on listing positive or negative aspects (39.1) and selecting the best aspect (41.3) (Grail et al., 2018). That profile is consistent with the view that OpinionQA becomes harder as models must aggregate and compare multiple latent opinions rather than classify a single polarity.

Generative review QA pushes the formulation further by asking for fluent opinion-aware answers. “Opinion-aware Answer Generation for Review-driven Question Answering in E-Commerce” proposes OAAG, which jointly learns answer generation and opinion mining, transfers the learned opinion structure through static or dynamic opinion fusion, and decodes with a multi-view pointer-generator network (Deng et al., 2020). On Electronics, OAAG-D reports R1 15.9, RL 14.5, B1 13.5, ES 85.7, and TOA 61.8, compared with RAGE at TOA 56.3; the paper emphasizes about a 6% gain in TOA overall (Deng et al., 2020). Here the defining problem is no longer only retrieval or extraction, but generating a response consistent with the common opinion in the evidence.

4. Survey-grounded OpinionQA and distributional alignment

Later work uses OpinionQA in a narrower sense: a benchmark built from Pew survey data, where the task is to model group-conditioned answers. One line of work describes it as a dataset inherited from Santurkar et al., with 100 questions sampled from a more contentious 500-question subset, and demographic groups such as Democrat, Republican, Male, Female, Black, and White (Meister et al., 2024). Another describes the broader resource as 1,498 multiple-choice survey questions from Pew Research’s American Trends Panel, yielding 91K unique <question, demographic> pairs (Zhang et al., 5 Oct 2025).

This survey-grounded setting changes the target from retrieving supporting review evidence to predicting either the majority answer for a group or the full answer distribution. “Benchmarking Distributional Alignment of LLMs” formalizes the distributional objective with average total variation distance:

$$\mathcal{A}(Y,\hat{Y}_{S, \mathcal{O}})= \frac{1}{|G|} \sum_{g \in G} \frac{1}{|Q|} \sum_{q \in Q} \frac{1}{2} {\left \lvert\lvert y_{g, q}- \hat{y}_{g, q} \right \rvert\rvert}_1 .$$

Lower values indicate better alignment between the gold group distribution and the model-predicted distribution (Meister et al., 2024). That paper varies three variables—question domain, steering method, and distribution expression method—and finds that direct verbalization of distributions is substantially better than token-probability extraction. In the combined leaderboard, GPT-4 (Verbalize) reaches 0.204 ± 0.004, the human baseline is 0.247 ± 0.004, the uniform distribution baseline is 0.302 ± 0.005, and GPT-4 (Log-p) is 0.582 ± 0.006 (Meister et al., 2024). The paper’s strongest methodological warning is that first-token log-probabilities can be worse than uniform for subgroup opinion distributions.

A related question is whether a model can infer an individual’s opinion from persona information. “Can LLM be a Personalized Judge?” evaluates exactly that and finds modest overall reliability in the OpinionQA setting: GPT-4 reaches 0.635 agreement overall, but 0.804 (385/480) on high-certainty samples and 0.602 (1526/2535) on low-certainty samples (Dong et al., 2024). The paper attributes this to persona sparsity and argues that uncertainty-aware abstention is more viable than unconditional prediction (Dong et al., 2024).

A third line reframes OpinionQA as a steerable pluralistic alignment task. “Exploring Chain-of-Thought Reasoning for Steerable Pluralistic Alignment” asks the model to answer a survey item from the perspective of a specified demographic group and evaluates Accuracy, Class-Balanced Accuracy, and Macro F1 (Zhang et al., 5 Oct 2025). Its best OpinionQA result is Llama 3 8B RLVR at 72.3 Acc / 74.5 BAcc / 68.4 MaF, outperforming the same model with standard supervised fine-tuning at 67.7 / 67.5 / 63.1 (Zhang et al., 5 Oct 2025). The same paper reports that RLVR reaches comparable validation accuracy with only 25–30% of the OpinionQA training data for Llama 3 8B (Zhang et al., 5 Oct 2025).

5. Plurality, aggregation, and opinion structure

Not all OpinionQA systems aim to collapse evidence into one answer. A substantial adjacent literature treats the output as a structured aggregation of viewpoints. “Query-Focused Opinion Summarization for User-Generated Content” formulates the task as selecting a summary set $S$ that jointly optimizes relevance, topic coverage, authorship coverage, polarity coverage, content coverage, and dispersion:

$$\mathcal{F}(S)=r(S)+\alpha t(S) +\beta a(S) +\gamma p(S) + \eta c(S) + \delta h(S).$$

The framework is explicitly designed for cases where a good answer must preserve diverse viewpoints and reduce redundancy rather than output a single canonical statement (Wang et al., 2016).

A more recent reformulation is QANA, which rewrites comments as questions, builds a weighted bipartite question-comment graph using embedding cosine similarity,

$$M(d_i, q_j) = \frac{\operatorname{emb}(d_i) \cdot \operatorname{emb}(q_j)}{|\operatorname{emb}(d_i)||\operatorname{emb}(q_j)|},$$

and ranks question nodes by centrality (Fukuma et al., 2024). In its key-point-generation setting, the best configuration covers 85% of human key points in the top 10 questions using GPT-4 question generation, text-embedding-3-large, and PageRank (Fukuma et al., 2024). This reframes opinion mining as discovering central answerable facets rather than directly predicting a single response.

Collective preference aggregation provides yet another perspective. “Collective Constitutional AI” does not predict survey answers directly; instead, it aggregates public input into a constitution using group-aware consensus,

$$\text{GAC}(s) = \prod_{g \in G} \text{P}(\text{agree}|g,s),$$

and then aligns a LLM to that constitution (Huang et al., 2024). Its relation to OpinionQA is cautionary. On OpinionQA, the Public and Standard models do not significantly differ in relative representativeness across ideologies, and the Public model is about 2–3 percentage points less representative overall; the paper hypothesizes that stronger emphasis on objectivity and impartiality leads to more refusals on subjective questions (Huang et al., 2024). The implication is that consensus-oriented public alignment and empirical opinion prediction are distinct objectives.

6. Evaluation, misconceptions, and open problems

Evaluation is a central unresolved problem because opinion answers can be fluent while still being wrong about aspect or sentiment. “OpinSummEval” makes this explicit for opinion summarization by collecting human ratings on Aspect Relevance, Self-Coherence, Sentiment Consistency, and Readability across outputs from 14 opinion summarization models (Shen et al., 2023). Its most important result is that ROUGE is unreliable in this setting, while BARTScore$_{rev \rightarrow hyp}$ is the strongest overall automatic metric, with system-level Kendall’s $\tau$ of 0.65 for Aspect Relevance, 0.76 for Self-Coherence, 0.77 for Sentiment Consistency, and 0.46 for Readability (Shen et al., 2023). This is directly relevant to OpinionQA: a model should be evaluated against source-grounded aspect-sentiment fidelity, not lexical overlap alone.

Several recurring limitations also define the field. SubjQA openly reports noise from its automatically constructed opinion KB, with only 35–50% mean precision across domains and about 48% of a random sample of question-review pairs marked unanswerable because of unrelated opinion neighbors (Bjerva et al., 2020). QANA depends heavily on question quality and approximates answerability with cosine similarity rather than explicit entailment or stance (Fukuma et al., 2024). Personalized judging is limited by persona sparsity, so even strong models only become reliable on a filtered high-certainty subset (Dong et al., 2024). Collective constitutional alignment can suppress minority-specific or polarizing preferences because it selects statements that survive cross-group consensus rather than preserving subgroup-specific constitutions (Huang et al., 2024). RLVR-based pluralistic alignment improves final OpinionQA accuracy but produces chains of thought that are less answer-recoverable, with 70.3% consistency on OpinionQA in the paper’s faithfulness test (Zhang et al., 5 Oct 2025).

A second common misconception is that opinion prediction, opinion summarization, and pluralistic alignment are interchangeable. The literature does not support that equivalence. Review-based OpinionQA often requires indirect evidence retrieval and question-conditioned stance aggregation (McAuley et al., 2015, Deng et al., 2020); distributional alignment requires calibrated subgroup distributions and is sensitive to how those distributions are elicited (Meister et al., 2024); public-input alignment may improve bias profiles without improving representativeness of survey responses (Huang et al., 2024). This suggests that OpinionQA is best treated as a family of related but technically distinct problems unified by perspective dependence, heterogeneous evidence, and irreducible disagreement.

The open problems are therefore structural rather than merely architectural. The literature repeatedly points to the need for span-level supervision combined with subjectivity labels, realistic unanswerable cases, multiple domains, mixed subjective and objective cases, demographic-conditional distributions rather than only majority labels, uncertainty-aware abstention, source-grounded evaluation, and mechanisms that preserve disagreement instead of collapsing it too early (Bjerva et al., 2020, Meister et al., 2024, Shen et al., 2023). In that sense, OpinionQA sits at the intersection of QA, opinion mining, personalization, and pluralistic alignment, and its difficulty is precisely that it requires models to answer from evidence without pretending that the evidence contains only one voice.