M-OS-PROMPTS Framework

• M-OS-PROMPTS is a framework for multi-source opinion summarization that combines subjective customer reviews with objective product metadata.
• It employs dual prompt families—M-OS-GEN-PROMPT for generating summaries and M-OS-EVAL-PROMPTS for structured, reference-free evaluation using LLMs as both architects and critics.
• Empirical findings show that the approach aligns closely with expert evaluations, improving summary quality and aiding user decision-making.

M-OS-PROMPTS is a prompt-based framework for multi-source opinion summarization (M-OS), a task that extends traditional opinion summarization by combining customer reviews with product metadata, especially the title, description, key features, specifications, and average rating, so that generated summaries capture both subjective opinion and objective product attributes. The framework assigns LLMs two complementary functions: architects, which generate summaries through M-OS-GEN-PROMPT, and critics, which evaluate summaries through M-OS-EVAL-PROMPTS. It is introduced together with M-OS-EVAL, a human-annotated, reference-free benchmark designed to test whether automated evaluators align with expert judgments across seven quality dimensions (Attri et al., 7 Jul 2025).

1. Task formulation and motivation

M-OS is presented as a richer formulation than classic opinion summarization. In the review-only setting, a model compresses subjective content from customer reviews, but product choice in e-commerce depends on more than sentiment alone. The M-OS formulation therefore expands the input space beyond reviews to include metadata, allowing a summary to report not only what users liked or disliked, but also what the product is and which concrete attributes define it. The paper illustrates this with an example in which a multi-source summary can include technical specifications such as 79g, head-heavy, 26.7 inches tall, and 7 mm beam width, whereas a review-only summary mainly captures broad praise and caveats (Attri et al., 7 Jul 2025).

The stated rationale is both practical and semantic. Reviews are subjective and often incomplete; metadata supplies the objective scaffolding that makes summaries more informative and less ambiguous. The paper attributes distinct roles to different metadata fields: descriptions identify the product and intended use, key features surface main selling points, specifications provide precise factual detail, and ratings provide an aggregate sentiment signal. In this formulation, a summary is intended to reduce decision fatigue and manual metadata parsing by presenting relevant facts and opinions in one coherent narrative.

A further motivation concerns evaluation. The paper states that traditional automatic metrics such as ROUGE, BERTScore, and BARTScore do not correlate well with human judgment for opinion summarization and are too narrow for the multi-source setting. This establishes M-OS not only as a generation problem but also as an evaluation problem requiring finer-grained, reference-free judgment.

2. Prompt architecture and the roles of LLMs

The proposed framework consists of two prompt families. M-OS-GEN-PROMPT is used for generation and instructs an LLM to synthesize information from the product title, description, key features, specifications, customer reviews, and average ratings into a comprehensive multi-source opinion summary. M-OS-EVAL-PROMPTS is used for evaluation and asks an LLM to assess a generated summary against the metadata and reviews along multiple dimensions (Attri et al., 7 Jul 2025).

This design is framed through the distinction between LLMs as architects and critics. As architects, LLMs perform synthesis over heterogeneous product sources. As critics, LLMs perform structured reference-free evaluation. The paper emphasizes that this dual use is central to the framework: the same model family is deployed both to generate summaries integrating many evidence sources and to judge those summaries with criteria intended to resemble human evaluation more closely than conventional lexical-overlap metrics.

The evaluation prompts are built from four recurring components: System Message, Task Description, Evaluation Criteria, and Evaluation Step. This turns evaluation into a structured reasoning procedure rather than a single unconstrained rating. The framework includes two evaluation prompt designs. Omni-Prompt is a metric-independent prompt that can be adapted across dimensions by changing only the metric component. Spectra-Prompts is a set of dimension-specific prompts, one for each evaluation dimension, intended to capture the particular nuance of each criterion. The paper argues that this structured prompt design helps prevent score inflation and encourages systematic inspection of summary content before a score is assigned.

3. M-OS-EVAL benchmark and annotation protocol

Because M-OS is introduced as a new task, the paper also introduces M-OS-EVAL, a reference-free evaluation benchmark for multi-source opinion summarization. Its purpose is to support meta-evaluation: automated evaluators, especially prompt-based LLM evaluators, are assessed according to how well they track expert human judgments (Attri et al., 7 Jul 2025).

The benchmark contains 14 model-generated summaries per product, 50 products from the M-OS-DATA test set, 3 expert raters per summary, 5-point Likert ratings, 7 evaluation dimensions, and 14,700 total ratings, computed as $3 \times 50 \times 14 \times 7$. The use of expert annotators rather than crowd workers is explicitly motivated by prior work indicating that expert judgments are more reliable for summary evaluation.

The annotation procedure has two rounds. When initial scores differ by 2 or more points, raters revisit the case and discuss it until disagreement is reduced to at most 1 point. Under this procedure, Krippendorff’s alpha rises from 0.70 in Round I to 0.86 in Round II. The benchmark therefore serves not only as a dataset of preferences, but also as a controlled protocol for stabilizing expert judgments.

The seven evaluation dimensions are the following:

Dimension	Criterion
Fluency	Grammaticality and readability
Coherence	Logical flow and organization
Relevance	Useful, product-relevant information
Faithfulness	Grounding in metadata and reviews, without hallucination
Aspect coverage	Coverage of important product aspects
Sentiment consistency	Alignment of summary sentiment with review evidence
Specificity	Concrete, non-generic details

These dimensions reflect the paper’s claim that summary quality in M-OS is multi-criteria. A fluent summary is insufficient if it omits technical facts, invents unsupported claims, or collapses diverse evidence into generic praise.

4. Scoring methodology and structured meta-evaluation

The paper adopts a probabilistic scoring procedure from prior work for prompt-based evaluation. If $s_k$ denotes a discrete score and $p(s_k)$ denotes the model-estimated probability of that score, the output score is computed as

$o = \sum_{k=1}^{j} p(s_k)\times s_k .$

The paper notes that $p(s_k)$ is approximated by sampling many outputs per input, with $n \approx 100$. This is intended to reduce the brittleness of one-shot LLM ratings and to provide a softer estimate of evaluator belief (Attri et al., 7 Jul 2025).

For summary-level comparison with human judgment, the paper uses a correlation-based meta-evaluation: $$\mathcal{R}(a,b) = \frac{1}{\mathcal{Q}} \sum_i \mathcal{K}\left( [\mathcal{M}_a(s_{i1}), \ldots, \mathcal{M}_a(s_{i\mathcal{N}})], [\mathcal{M}_b(s_{i1}), \ldots, \mathcal{M}_b(s_{i\mathcal{N}})] \right),$$ where $\mathcal{Q}$ is the number of products, $s_{ij}$ is the $j$-th summary for product $s_k$0, $s_k$1 and $s_k$2 are evaluation metrics, and $s_k$3 is the correlation measure. This formalizes evaluator quality as agreement in relative ranking, not merely raw score similarity.

Within this setup, the paper distinguishes metric-independent and metric-dependent prompting. Omni-Prompt is the metric-independent design, whereas Spectra-Prompts is the metric-dependent design. The authors argue that evaluator prompts should not simply ask whether a summary is good; instead they should require the model to identify summary elements, analyze them against explicit criteria, and assign a score based on defined ranges. This suggests that evaluator performance depends not only on the underlying LLM but also on how strongly the prompt enforces criterion-specific reasoning.

5. Empirical findings on generation and evaluation

The experiments assess both summary generation and automated evaluation. For generation, the paper reports that large LLMs perform best as summary generators, with Qwen2.5-72B-Instruct achieving the highest overall human score at 4.186, followed closely by GPT-4o at 4.169. This is presented as evidence that large-context, instruction-tuned LLMs are well suited to synthesizing long, heterogeneous product inputs (Attri et al., 7 Jul 2025).

For evaluation, the principal result is alignment with expert judgment. In the metric-dependent setting, Spectra-Prompts outperform earlier prompt-based baselines. In the metric-independent setting, Omni-Prompt with GPT-4o achieves the strongest overall alignment, with an average Spearman correlation of $s_k$4 across the seven dimensions. The paper identifies this as the headline automatic-evaluation result for M-OS-PROMPTS.

The paper interprets this alignment as a consequence of structured prompt design. Baseline prompts are said to inflate scores, whereas the proposed evaluator prompts enforce a more disciplined procedure by requiring the model to inspect evidence, reason through the criteria, and then score. The resulting evaluator is therefore not merely a generic judge; it is a prompt-programmed assessor for a multi-criteria, evidence-grounded summarization task.

6. User preference, interpretive claims, and broader significance

The user study tests whether metadata-enriched summaries are actually preferred in use. Using the top generation model, Qwen2.5-72B-Instruct, the study includes 300 participants aged 18–50 who compare anonymized M-OS summaries with traditional opinion summaries across five evaluation criteria. On average, 87% of participants preferred the M-OS summaries. The paper presents this as evidence that factually enriched summaries improve user engagement (Attri et al., 7 Jul 2025).

A central interpretive claim follows from this result: the most useful product summaries are not those that merely sound fluent, but those that are grounded in evidence, rich in relevant detail, faithful to multiple sources, sentiment-consistent, and specific enough to support comparison and purchase decisions. This directly opposes a common misconception that opinion summarization quality can be judged primarily by readability or stylistic polish.

The broader significance of M-OS-PROMPTS lies in the integration of task formulation, prompt design, and evaluation methodology. The framework treats summarization as a multi-source and multi-criteria problem, and it treats evaluation as a structured reasoning task rather than a lexical-overlap calculation. The benchmark, the generation prompt, and the evaluation prompts therefore form a single pipeline: metadata-enhanced generation, expert-grounded reference-free evaluation, and meta-evaluation against human judgment. In that sense, M-OS-PROMPTS is positioned not only as a method for producing better product summaries, but also as an effort to define what quality means for multi-source opinion summarization.