---
title: 'OpinionQA: Subjective Question Answering'
url: https://www.emergentmind.com/topics/opinionqa
type: topic
---

# OpinionQA: Subjective Question Answering

OpinionQA denotes a family of question-answering problems in which the target is not a single objectively correct fact but an opinion, experience, judgment, or demographic-conditioned response distribution. In the review-grounded literature, the evidence comes from customer reviews and community answers, and the central difficulties are ambiguity, subjectivity, and diverging viewpoints; in later survey-grounded work, OpinionQA becomes a benchmark for predicting how a specified demographic group answers a multiple-choice question or how its full answer distribution is expressed [1610.08095][2411.05403]. Across these formulations, the common requirement is to model answers that are conditional on perspective rather than uniquely determined by an external fact.

## 1. Conceptual foundations

The review-grounded formulation of OpinionQA emerged from the observation that many real queries are not factoid queries at all. Product questions such as whether an item is “loud enough,” “vegan friendly,” or suitable for a particular use case are grounded in user experiences and private states rather than encyclopedic facts [1512.06863][2004.14283]. In this setting, the standard QA assumption of a single correct answer becomes unstable.

A canonical statement of the problem is that opinion question answering differs from traditional QA in at least two ways: questions may have “multiple, often divergent, answers,” and what counts as a good answer can depend on the asker and the answerer [1610.08095]. SubjQA makes the same point in a different vocabulary, defining subjectivity as the expression of internal opinions or beliefs “which cannot be objectively observed or verified,” and showing that subjectivity matters in QA with “more intricate interactions” than in sentiment classification [2004.14283].

A frequent misconception is that OpinionQA is merely domain transfer from Wikipedia-style reading comprehension to reviews. SubjQA explicitly rejects that interpretation: a subjective question may or may not have a subjective answer, and a factual question can occur in a review context saturated with subjective language [2004.14283]. This suggests that OpinionQA is not just QA over a different corpus; it is QA under perspective dependence, mixed evidence, and answer pluralism.

## 2. Review-grounded corpora and task formulations

An early large-scale formulation appears in “Modeling Ambiguity, Subjectivity, and Diverging Viewpoints in Opinion Question Answering Systems” [1610.08095]. That work argued that review QA must explicitly represent ambiguity and personalization, and reported a dataset with **800 thousand questions—and over 3.1 million answers** [1610.08095].

Representative resources span retrieval-oriented, extractive, projective, generative, and survey-grounded settings.

| Resource | Evidence and scale | Main formulation |
|---|---|---|
| “Moqa” [1512.06863] | **1,447,173 questions with answers** and **13,498,681 customer reviews** | Reviews as experts for binary and open-ended product queries |
| “ReviewQA” [1810.12196] | **587,492 questions** over **100,000 hotel reviews** | Projective aspect-based QA with **8 task groups** |
| “SubjQA” [2004.14283] | **over 10,000 English examples** across **6 distinct domains** | Extractive QA with subjectivity labels for questions and answer spans |
| “OpinionQA” [2510.04045] | **1,498 multiple-choice survey questions** and **91K unique `<question, demographic>` pairs** | Demographic-conditioned majority-answer prediction |

The review-grounded benchmarks emphasize different structural properties. ReviewQA organizes hotel-review QA around aspect detection, aspect-specific satisfaction, threshold comparison, exact rating prediction, aspect comparison, and best/worst aspect selection; its answer space is projective rather than extractive [1810.12196]. SubjQA instead focuses on extractive review comprehension with explicit subjectivity labels, span supervision, and realistic unanswerability [2004.14283].

SubjQA is particularly informative about the statistical structure of opinion-oriented QA. It reports **73% subjective questions** and **74% subjective answers**, with roughly **65%** of question-review pairs answerable overall [2004.14283]. Its reasoning analysis of 100 sampled answerable development questions assigns **18%** to lexical reasoning, **28%** to paraphrase, **43%** to indirect reasoning, and **11%** to noisy or non-applicable cases [2004.14283]. Those numbers formalize a core property of OpinionQA: much of the signal is indirect rather than lexically aligned.

## 3. Modeling architectures

A foundational model for review-based OpinionQA is **Moqa**, a mixture-of-experts framework in which each review is treated as an expert [1512.06863]. Its generic prediction rule is

$$
p(y \mid X) = \sum_f p(f \mid X)\, p(y \mid f, X),
$$

where the expert is a review, the gate is a learned relevance function, and the expert vote is a learned answer-support function [1512.06863]. Moqa is designed for heterogeneous linguistic mappings between question language and review language, using BM25+, ROUGE-L, and a bilinear compatibility term in its relevance model [1512.06863]. On binary yes/no questions, it reports **76.8%** average **accuracy@50%** across the eight largest categories; on open-ended questions, it reports **0.883** average **AUC** [1512.06863].

SubjQA introduces a different modeling question: whether explicit subjectivity supervision improves extractive QA. Its multitask subjectivity-aware model extends FastQA with a shared encoder and a subjectivity classification head trained jointly with span prediction [2004.14283]. The paper reports that out-of-the-box SQuAD-pretrained models reach only **30.5% F1** on average across SubjQA domains, while fine-tuned BERT reaches **74.1% F1**, and the multitask subjectivity-aware model reaches **76.3% F1** [2004.14283]. The broader implication is that subjectivity is not merely metadata; it is useful supervision for answer extraction itself.

ReviewQA, by contrast, is designed to test reasoning over latent aspect structure rather than span extraction. Its strongest baseline, a deep projective reader, reaches **60.4** overall accuracy, with especially strong results on overall satisfaction (**90.9**), aspect sentiment (**85.9**), and threshold comparison (**91.3**), but much weaker performance on listing positive or negative aspects (**39.1**) and selecting the best aspect (**41.3**) [1810.12196]. That profile is consistent with the view that OpinionQA becomes harder as models must aggregate and compare multiple latent opinions rather than classify a single polarity.

Generative review QA pushes the formulation further by asking for fluent opinion-aware answers. “Opinion-aware Answer Generation for Review-driven Question Answering in E-Commerce” proposes **OAAG**, which jointly learns answer generation and opinion mining, transfers the learned opinion structure through static or dynamic opinion fusion, and decodes with a multi-view pointer-generator network [2008.11972]. On Electronics, **OAAG-D** reports **R1 15.9**, **RL 14.5**, **B1 13.5**, **ES 85.7**, and **TOA 61.8**, compared with **RAGE** at **TOA 56.3**; the paper emphasizes about a **6% gain in TOA** overall [2008.11972]. Here the defining problem is no longer only retrieval or extraction, but generating a response consistent with the common opinion in the evidence.

## 4. Survey-grounded OpinionQA and distributional alignment

Later work uses **OpinionQA** in a narrower sense: a benchmark built from Pew survey data, where the task is to model group-conditioned answers. One line of work describes it as a dataset inherited from Santurkar et al., with **100 questions** sampled from a more contentious **500-question subset**, and demographic groups such as **Democrat, Republican, Male, Female, Black, and White** [2411.05403]. Another describes the broader resource as **1,498 multiple-choice survey questions** from Pew Research’s American Trends Panel, yielding **91K unique `<question, demographic>` pairs** [2510.04045].

This survey-grounded setting changes the target from retrieving supporting review evidence to predicting either the majority answer for a group or the full answer distribution. “Benchmarking Distributional Alignment of Large Language Models” formalizes the distributional objective with average total variation distance:

$$
\mathcal{A}(Y,\hat{Y}_{S, \mathcal{O}})= \frac{1}{|G|} \sum_{g \in G} \frac{1}{|Q|} \sum_{q \in Q} \frac{1}{2} {\left \lvert\lvert y_{g, q}- \hat{y}_{g, q} \right \rvert\rvert}_1 .
$$

Lower values indicate better alignment between the gold group distribution and the model-predicted distribution [2411.05403]. That paper varies three variables—**question domain**, **steering method**, and **distribution expression method**—and finds that direct verbalization of distributions is substantially better than token-probability extraction. In the combined leaderboard, **GPT-4 (Verbalize)** reaches **0.204 ± 0.004**, the **human** baseline is **0.247 ± 0.004**, the **uniform distribution** baseline is **0.302 ± 0.005**, and **GPT-4 (Log-p)** is **0.582 ± 0.006** [2411.05403]. The paper’s strongest methodological warning is that first-token log-probabilities can be worse than uniform for subgroup opinion distributions.

A related question is whether a model can infer an individual’s opinion from persona information. “Can LLM be a Personalized Judge?” evaluates exactly that and finds modest overall reliability in the OpinionQA setting: **GPT-4** reaches **0.635** agreement overall, but **0.804 (385/480)** on high-certainty samples and **0.602 (1526/2535)** on low-certainty samples [2406.11657]. The paper attributes this to **persona sparsity** and argues that uncertainty-aware abstention is more viable than unconditional prediction [2406.11657].

A third line reframes OpinionQA as a steerable pluralistic alignment task. “Exploring Chain-of-Thought Reasoning for Steerable Pluralistic Alignment” asks the model to answer a survey item from the perspective of a specified demographic group and evaluates **Accuracy**, **Class-Balanced Accuracy**, and **Macro F1** [2510.04045]. Its best OpinionQA result is **Llama 3 8B RLVR** at **72.3 Acc / 74.5 BAcc / 68.4 MaF**, outperforming the same model with standard supervised fine-tuning at **67.7 / 67.5 / 63.1** [2510.04045]. The same paper reports that RLVR reaches comparable validation accuracy with only **25–30%** of the OpinionQA training data for Llama 3 8B [2510.04045].

## 5. Plurality, aggregation, and opinion structure

Not all OpinionQA systems aim to collapse evidence into one answer. A substantial adjacent literature treats the output as a structured aggregation of viewpoints. “Query-Focused Opinion Summarization for User-Generated Content” formulates the task as selecting a summary set \(S\) that jointly optimizes relevance, topic coverage, authorship coverage, polarity coverage, content coverage, and dispersion:

$$
\mathcal{F}(S)=r(S)+\alpha t(S) +\beta a(S) +\gamma p(S) + \eta c(S) + \delta h(S).
$$

The framework is explicitly designed for cases where a good answer must preserve diverse viewpoints and reduce redundancy rather than output a single canonical statement [1606.05702].

A more recent reformulation is **QANA**, which rewrites comments as questions, builds a weighted bipartite question-comment graph using embedding cosine similarity,

$$
M(d_i, q_j) = \frac{\operatorname{emb}(d_i) \cdot \operatorname{emb}(q_j)}{|\operatorname{emb}(d_i)||\operatorname{emb}(q_j)|},
$$

and ranks question nodes by centrality [2404.18371]. In its key-point-generation setting, the best configuration covers **85% of human key points in the top 10 questions** using **GPT-4** question generation, **text-embedding-3-large**, and **PageRank** [2404.18371]. This reframes opinion mining as discovering central answerable facets rather than directly predicting a single response.

Collective preference aggregation provides yet another perspective. “Collective Constitutional AI” does not predict survey answers directly; instead, it aggregates public input into a constitution using **group-aware consensus**,

$$
\text{GAC}(s) = \prod_{g \in G} \text{P}(\text{agree}|g,s),
$$

and then aligns a language model to that constitution [2406.07814]. Its relation to OpinionQA is cautionary. On OpinionQA, the **Public** and **Standard** models do not significantly differ in relative representativeness across ideologies, and the **Public** model is about **2–3 percentage points** less representative overall; the paper hypothesizes that stronger emphasis on objectivity and impartiality leads to more refusals on subjective questions [2406.07814]. The implication is that consensus-oriented public alignment and empirical opinion prediction are distinct objectives.

## 6. Evaluation, misconceptions, and open problems

Evaluation is a central unresolved problem because opinion answers can be fluent while still being wrong about aspect or sentiment. “OpinSummEval” makes this explicit for opinion summarization by collecting human ratings on **Aspect Relevance**, **Self-Coherence**, **Sentiment Consistency**, and **Readability** across outputs from **14 opinion summarization models** [2310.18122]. Its most important result is that **ROUGE** is unreliable in this setting, while **BARTScore\(_{rev \rightarrow hyp}\)** is the strongest overall automatic metric, with system-level Kendall’s \(\tau\) of **0.65** for Aspect Relevance, **0.76** for Self-Coherence, **0.77** for Sentiment Consistency, and **0.46** for Readability [2310.18122]. This is directly relevant to OpinionQA: a model should be evaluated against source-grounded aspect-sentiment fidelity, not lexical overlap alone.

Several recurring limitations also define the field. SubjQA openly reports noise from its automatically constructed opinion KB, with only **35–50% mean precision** across domains and about **48%** of a random sample of question-review pairs marked unanswerable because of unrelated opinion neighbors [2004.14283]. QANA depends heavily on question quality and approximates answerability with cosine similarity rather than explicit entailment or stance [2404.18371]. Personalized judging is limited by persona sparsity, so even strong models only become reliable on a filtered high-certainty subset [2406.11657]. Collective constitutional alignment can suppress minority-specific or polarizing preferences because it selects statements that survive cross-group consensus rather than preserving subgroup-specific constitutions [2406.07814]. RLVR-based pluralistic alignment improves final OpinionQA accuracy but produces chains of thought that are less answer-recoverable, with **70.3%** consistency on OpinionQA in the paper’s faithfulness test [2510.04045].

A second common misconception is that opinion prediction, opinion summarization, and pluralistic alignment are interchangeable. The literature does not support that equivalence. Review-based OpinionQA often requires indirect evidence retrieval and question-conditioned stance aggregation [1512.06863][2008.11972]; distributional alignment requires calibrated subgroup distributions and is sensitive to how those distributions are elicited [2411.05403]; public-input alignment may improve bias profiles without improving representativeness of survey responses [2406.07814]. This suggests that OpinionQA is best treated as a family of related but technically distinct problems unified by perspective dependence, heterogeneous evidence, and irreducible disagreement.

The open problems are therefore structural rather than merely architectural. The literature repeatedly points to the need for span-level supervision combined with subjectivity labels, realistic unanswerable cases, multiple domains, mixed subjective and objective cases, demographic-conditional distributions rather than only majority labels, uncertainty-aware abstention, source-grounded evaluation, and mechanisms that preserve disagreement instead of collapsing it too early [2004.14283][2411.05403][2310.18122]. In that sense, OpinionQA sits at the intersection of QA, opinion mining, personalization, and pluralistic alignment, and its difficulty is precisely that it requires models to answer from evidence without pretending that the evidence contains only one voice.

Source: https://www.emergentmind.com/topics/opinionqa