---
title: M-OS-PROMPTS Framework
url: https://www.emergentmind.com/topics/m-os-prompts
type: topic
---

# M-OS-PROMPTS Framework

M-OS-PROMPTS is a prompt-based framework for **multi-source opinion summarization (M-OS)**, a task that extends traditional opinion summarization by combining customer reviews with product metadata, especially the **title, description, key features, specifications, and average rating**, so that generated summaries capture both subjective opinion and objective product attributes. The framework assigns large language models (LLMs) two complementary functions: **architects**, which generate summaries through **M-OS-GEN-PROMPT**, and **critics**, which evaluate summaries through **M-OS-EVAL-PROMPTS**. It is introduced together with **M-OS-EVAL**, a human-annotated, reference-free benchmark designed to test whether automated evaluators align with expert judgments across seven quality dimensions [2507.04751].

## 1. Task formulation and motivation

M-OS is presented as a richer formulation than classic opinion summarization. In the review-only setting, a model compresses subjective content from customer reviews, but product choice in e-commerce depends on more than sentiment alone. The M-OS formulation therefore expands the input space beyond reviews to include metadata, allowing a summary to report not only what users liked or disliked, but also what the product is and which concrete attributes define it. The paper illustrates this with an example in which a multi-source summary can include technical specifications such as **79g**, **head-heavy**, **26.7 inches tall**, and **7 mm beam width**, whereas a review-only summary mainly captures broad praise and caveats [2507.04751].

The stated rationale is both practical and semantic. Reviews are subjective and often incomplete; metadata supplies the objective scaffolding that makes summaries more informative and less ambiguous. The paper attributes distinct roles to different metadata fields: **descriptions** identify the product and intended use, **key features** surface main selling points, **specifications** provide precise factual detail, and **ratings** provide an aggregate sentiment signal. In this formulation, a summary is intended to reduce decision fatigue and manual metadata parsing by presenting relevant facts and opinions in one coherent narrative.

A further motivation concerns evaluation. The paper states that traditional automatic metrics such as **ROUGE, BERTScore, and BARTScore** do not correlate well with human judgment for opinion summarization and are too narrow for the multi-source setting. This establishes M-OS not only as a generation problem but also as an evaluation problem requiring finer-grained, reference-free judgment.

## 2. Prompt architecture and the roles of LLMs

The proposed framework consists of two prompt families. **M-OS-GEN-PROMPT** is used for generation and instructs an LLM to synthesize information from the **product title, description, key features, specifications, customer reviews, and average ratings** into a comprehensive multi-source opinion summary. **M-OS-EVAL-PROMPTS** is used for evaluation and asks an LLM to assess a generated summary against the metadata and reviews along multiple dimensions [2507.04751].

This design is framed through the distinction between LLMs as **architects** and **critics**. As architects, LLMs perform synthesis over heterogeneous product sources. As critics, LLMs perform structured reference-free evaluation. The paper emphasizes that this dual use is central to the framework: the same model family is deployed both to generate summaries integrating many evidence sources and to judge those summaries with criteria intended to resemble human evaluation more closely than conventional lexical-overlap metrics.

The evaluation prompts are built from four recurring components: **System Message**, **Task Description**, **Evaluation Criteria**, and **Evaluation Step**. This turns evaluation into a structured reasoning procedure rather than a single unconstrained rating. The framework includes two evaluation prompt designs. **Omni-Prompt** is a metric-independent prompt that can be adapted across dimensions by changing only the metric component. **Spectra-Prompts** is a set of dimension-specific prompts, one for each evaluation dimension, intended to capture the particular nuance of each criterion. The paper argues that this structured prompt design helps prevent score inflation and encourages systematic inspection of summary content before a score is assigned.

## 3. M-OS-EVAL benchmark and annotation protocol

Because M-OS is introduced as a new task, the paper also introduces **M-OS-EVAL**, a **reference-free evaluation benchmark** for multi-source opinion summarization. Its purpose is to support meta-evaluation: automated evaluators, especially prompt-based LLM evaluators, are assessed according to how well they track expert human judgments [2507.04751].

The benchmark contains **14 model-generated summaries per product**, **50 products** from the M-OS-DATA test set, **3 expert raters** per summary, **5-point Likert ratings**, **7 evaluation dimensions**, and **14,700 total ratings**, computed as \(3 \times 50 \times 14 \times 7\). The use of expert annotators rather than crowd workers is explicitly motivated by prior work indicating that expert judgments are more reliable for summary evaluation.

The annotation procedure has two rounds. When initial scores differ by **2 or more points**, raters revisit the case and discuss it until disagreement is reduced to at most **1 point**. Under this procedure, **Krippendorff’s alpha** rises from **0.70** in Round I to **0.86** in Round II. The benchmark therefore serves not only as a dataset of preferences, but also as a controlled protocol for stabilizing expert judgments.

The seven evaluation dimensions are the following:

| Dimension | Criterion |
|---|---|
| Fluency | Grammaticality and readability |
| Coherence | Logical flow and organization |
| Relevance | Useful, product-relevant information |
| Faithfulness | Grounding in metadata and reviews, without hallucination |
| Aspect coverage | Coverage of important product aspects |
| Sentiment consistency | Alignment of summary sentiment with review evidence |
| Specificity | Concrete, non-generic details |

These dimensions reflect the paper’s claim that summary quality in M-OS is multi-criteria. A fluent summary is insufficient if it omits technical facts, invents unsupported claims, or collapses diverse evidence into generic praise.

## 4. Scoring methodology and structured meta-evaluation

The paper adopts a probabilistic scoring procedure from prior work for prompt-based evaluation. If \(s_k\) denotes a discrete score and \(p(s_k)\) denotes the model-estimated probability of that score, the output score is computed as
\[
o = \sum_{k=1}^{j} p(s_k)\times s_k .
\]
The paper notes that \(p(s_k)\) is approximated by sampling many outputs per input, with \(n \approx 100\). This is intended to reduce the brittleness of one-shot LLM ratings and to provide a softer estimate of evaluator belief [2507.04751].

For summary-level comparison with human judgment, the paper uses a correlation-based meta-evaluation:
\[
\mathcal{R}(a,b) = \frac{1}{\mathcal{Q}} \sum_i \mathcal{K}\left( [\mathcal{M}_a(s_{i1}), \ldots, \mathcal{M}_a(s_{i\mathcal{N}})], [\mathcal{M}_b(s_{i1}), \ldots, \mathcal{M}_b(s_{i\mathcal{N}})] \right),
\]
where \(\mathcal{Q}\) is the number of products, \(s_{ij}\) is the \(j\)-th summary for product \(p_i\), \(\mathcal{M}_a\) and \(\mathcal{M}_b\) are evaluation metrics, and \(\mathcal{K}\) is the correlation measure. This formalizes evaluator quality as agreement in relative ranking, not merely raw score similarity.

Within this setup, the paper distinguishes **metric-independent** and **metric-dependent** prompting. **Omni-Prompt** is the metric-independent design, whereas **Spectra-Prompts** is the metric-dependent design. The authors argue that evaluator prompts should not simply ask whether a summary is good; instead they should require the model to identify summary elements, analyze them against explicit criteria, and assign a score based on defined ranges. This suggests that evaluator performance depends not only on the underlying LLM but also on how strongly the prompt enforces criterion-specific reasoning.

## 5. Empirical findings on generation and evaluation

The experiments assess both summary generation and automated evaluation. For generation, the paper reports that large LLMs perform best as summary generators, with **Qwen2.5-72B-Instruct** achieving the highest overall human score at **4.186**, followed closely by **GPT-4o** at **4.169**. This is presented as evidence that large-context, instruction-tuned LLMs are well suited to synthesizing long, heterogeneous product inputs [2507.04751].

For evaluation, the principal result is alignment with expert judgment. In the metric-dependent setting, **Spectra-Prompts** outperform earlier prompt-based baselines. In the metric-independent setting, **Omni-Prompt with GPT-4o** achieves the strongest overall alignment, with an **average Spearman correlation of \(\rho = 0.74\)** across the seven dimensions. The paper identifies this as the headline automatic-evaluation result for M-OS-PROMPTS.

The paper interprets this alignment as a consequence of structured prompt design. Baseline prompts are said to inflate scores, whereas the proposed evaluator prompts enforce a more disciplined procedure by requiring the model to inspect evidence, reason through the criteria, and then score. The resulting evaluator is therefore not merely a generic judge; it is a prompt-programmed assessor for a multi-criteria, evidence-grounded summarization task.

## 6. User preference, interpretive claims, and broader significance

The user study tests whether metadata-enriched summaries are actually preferred in use. Using the top generation model, **Qwen2.5-72B-Instruct**, the study includes **300 participants** aged **18–50** who compare anonymized **M-OS summaries** with traditional opinion summaries across five evaluation criteria. On average, **87% of participants preferred the M-OS summaries**. The paper presents this as evidence that factually enriched summaries improve user engagement [2507.04751].

A central interpretive claim follows from this result: the most useful product summaries are not those that merely sound fluent, but those that are **grounded in evidence, rich in relevant detail, faithful to multiple sources, sentiment-consistent, and specific enough to support comparison and purchase decisions**. This directly opposes a common misconception that opinion summarization quality can be judged primarily by readability or stylistic polish.

The broader significance of M-OS-PROMPTS lies in the integration of task formulation, prompt design, and evaluation methodology. The framework treats summarization as a **multi-source** and **multi-criteria** problem, and it treats evaluation as a structured reasoning task rather than a lexical-overlap calculation. The benchmark, the generation prompt, and the evaluation prompts therefore form a single pipeline: metadata-enhanced generation, expert-grounded reference-free evaluation, and meta-evaluation against human judgment. In that sense, M-OS-PROMPTS is positioned not only as a method for producing better product summaries, but also as an effort to define what quality means for multi-source opinion summarization.

Source: https://www.emergentmind.com/topics/m-os-prompts