---
title: 'M-OS-EVAL: Benchmark for Multi-Source Opinion Summaries'
url: https://www.emergentmind.com/topics/m-os-eval
type: topic
---

# M-OS-EVAL: Benchmark for Multi-Source Opinion Summaries

Searching arXiv for papers directly relevant to “M-OS-EVAL” and adjacent evaluation methodology in multi-source opinion summarization.
M-OS-EVAL is a reference-free benchmark dataset for evaluating **multi-source opinion summaries** in e-commerce settings. It was introduced together with **Multi-Source Opinion Summarization (M-OS)**, a task that extends review-only opinion summarization by incorporating product metadata such as title, description, key features, specifications, and average ratings alongside customer reviews. Its central purpose is to provide a benchmark tailored to summaries that must jointly capture subjective opinions and objective product attributes, and to support the development of LLM-based evaluators that correlate with expert human judgment across seven quality dimensions: fluency, coherence, relevance, faithfulness, aspect coverage, sentiment consistency, and specificity [2507.04751].

## 1. Definition within multi-source opinion summarization

M-OS-EVAL is defined in the context of M-OS rather than traditional opinion summarization. In the underlying task, the input is not limited to customer reviews; it includes both **subjective evidence** from reviews and **objective product metadata** from titles, descriptions, key features, specifications, and ratings. The benchmark therefore targets summaries that integrate heterogeneous sources into a single output.

This task formulation addresses a specific limitation of review-only summarization: a summary may capture consensus sentiment while omitting concrete product attributes needed for product comparison and purchase decisions. M-OS-EVAL operationalizes evaluation for this richer setting by treating metadata-grounded summarization as a distinct problem rather than as a minor variant of conventional opinion summarization. A plausible implication is that evaluation criteria for M-OS must reward not only linguistic quality and sentiment fidelity but also correct integration of factual product information.

The benchmark is also motivated by the inadequacy of standard reference-based automatic metrics such as **ROUGE**, **BLEU**, **BERTSCORE**, and **BARTSCORE** for this setting. The benchmark treats M-OS evaluation as fundamentally **reference-free**, because metadata-enriched summaries may be factually richer and structurally different from any single reference while still being superior.

## 2. Dataset construction and annotation protocol

M-OS-EVAL is built on top of **M-OS-DATA**, a proprietary product-metadata dataset created through collaboration between the authors’ university lab and a major e-commerce company [2507.04751]. M-OS-DATA contains **7,752 unique queries** and **23,256 products**, with each product including title, description, features, specifications, reviews, and average rating. The reported averages are **10 reviews per product**, **242.6 words** of specifications, **17.99 words** of reviews, **105.79 words** of description, and **24.64 words** of key features.

The benchmark itself is constructed from the **M-OS-DATA test set**. It uses **50 products**, each paired with **14 model-generated summaries**. Every summary is evaluated on **7 dimensions**, and each summary-dimension pair is rated by **3 expert raters**. This yields **4,900 summary-dimension instances** and **14,700 total ratings**:
$$
3 \text{ raters} \times 50 \text{ products} \times 14 \text{ summaries} \times 7 \text{ dimensions} = 14{,}700.
$$

The 14 summaries per product come from 14 LLMs: **Mistral-7B-Instruct-v0.3**, **Meta-Llama-3.1-8B-Instruct**, **Mistral-7B-Instruct-v0.2**, **gemma-7b-it**, **vicuna-7b-v1.5**, **zephyr-7b-beta**, **GPT-4o**, **Gemma-2-9b-it**, **Mistral-Small-Instruct-2409**, **Mixtral-8x7B-Instruct-v0.1**, **Qwen2.5-7B-Instruct**, **Qwen2.5-32B-Instruct**, **Qwen2.5-72B-Instruct**, and **Meta-Llama-3.1-70B-Instruct**.

The human annotation protocol uses **3 experienced raters**: one Master’s-level, one Pre-Doctoral, and one Doctoral-level. All were male, aged 24–32, and had publications or active research in opinion summarization or were working in the domain. Annotation proceeded in **two rounds**. In **Round I**, ratings were assigned independently. In **Round II**, if ratings differed by **2 or more points**, raters re-evaluated through discussion until the discrepancy was reduced to **1 point or less**. Model identities were undisclosed to reduce bias.

| Component | Value | Notes |
|---|---:|---|
| Products | 50 | From the M-OS-DATA test set |
| Summaries per product | 14 | One from each evaluated LLM |
| Dimensions | 7 | Human and LLM-evaluator targets |
| Raters per item | 3 | Expert raters |
| Summary-dimension instances | 4,900 | \(50 \times 14 \times 7\) |
| Total ratings | 14,700 | \(3 \times 50 \times 14 \times 7\) |

## 3. Evaluation dimensions and agreement structure

M-OS-EVAL evaluates summaries along seven dimensions [2507.04751]. The paper points to an appendix for exact definitions, but the main text characterizes the intended focus of each dimension clearly enough to support practical use.

| Dimension | Evaluative focus |
|---|---|
| Fluency | Grammaticality, readability, and naturalness |
| Coherence | Logical flow and organization |
| Relevance | Inclusion of important product information |
| Faithfulness | Factual correctness relative to sources |
| Aspect Coverage | Coverage of important product aspects |
| Sentiment Consistency | Alignment with review sentiment |
| Specificity | Product-specific detail and precision |

Each dimension is scored on a **5-point Likert scale**. The annotation analysis reports **Krippendorff’s alpha** for both annotation rounds. Overall agreement rises from **0.70** in Round I to **0.86** in Round II. Dimension-wise values are: **Fluency 0.73 → 0.88**, **Coherence 0.67 → 0.82**, **Relevance 0.69 → 0.85**, **Faithfulness 0.79 → 0.91**, **Aspect Coverage 0.77 → 0.89**, **Sentiment Consistency 0.66 → 0.86**, and **Specificity 0.61 → 0.84**.

These values are interpreted in the benchmark using the reported ranges **0.61–0.80 = moderate** and **0.81–1.00 = substantial**. The increase from Round I to Round II indicates that the combination of expert raters, detailed guidelines, and discussion-based reconciliation materially improves consistency. A plausible implication is that dimensions tied to product metadata, especially faithfulness and aspect coverage, benefit from more explicit rubric standardization than dimensions driven by narrative judgment.

## 4. Prompt-based LLM evaluators

M-OS-EVAL is not only a human-annotated benchmark; it is also a testbed for **LLM-based evaluation prompts**. The paper introduces **M-OS-PROMPTS** for both generation and evaluation, with the evaluation side centered on **M-OS-EVAL-PROMPTS** [2507.04751].

Each evaluation prompt contains four components: a **System Message**, a **Task Description**, **Evaluation Criteria**, and an **Evaluation Step**. Two prompt families are defined. **Omni-Prompt** is **metric-independent** and uses a common structure across dimensions with a dynamic metric slot. **Spectra-Prompts** are **metric-dependent**, with one specialized prompt per dimension. These are compared against prior prompt-based evaluators from **Siledar et al.**: **Op-I-Prompt** and **Op-Prompts**.

The evaluator backbones include **Llama-3.1-8B-Instruct**, **Mistral-7B-Instruct-v0.2**, **Mistral-7B-Instruct-v0.3**, **Llama-3.1-70B-Instruct**, and **GPT-4o**. Scoring follows the weighted-average scheme associated with G-Eval:
$$
o = \sum_{k=1}^{j} p(s_k)\times s_k
$$
where \(s_k\) are possible scores and \(p(s_k)\) are their LLM-determined probabilities. The paper states that \(p(s_k)\) is estimated by sampling with \(n \approx 100\) outputs per input.

Meta-evaluation uses summary-level correlation following **Bhandari et al.**, with **Spearman \((\rho)\)** and **Kendall Tau \((\tau)\)** as the reported correlation measures. The benchmark therefore serves two roles simultaneously: it is a dataset of expert judgments over M-OS summaries, and it is an evaluation target for prompt-engineered LLM judges.

## 5. Reported empirical findings

The benchmark supports two main experimental lines: evaluation of M-OS generation systems and evaluation of LLM judges [2507.04751].

For human evaluation of generated summaries, the best overall model is **Qwen2.5-72B-Instruct** with an average score of **4.186**, followed by **GPT-4o** at **4.169** and **Qwen2.5-32B-Instruct** at **4.149**. The reported pattern is that most models score highly on fluency, while larger differences appear on specificity. Open-source models are described as competitive with GPT-4o, and **Qwen2.5-72B-Instruct** is reported as outperforming GPT-4o overall.

For meta-evaluation, **Spectra-GPT-4o** achieves the best average **Spearman correlation of 0.70** among metric-dependent prompts. In the metric-independent setting, **Omni-Prompt + GPT-4o** reaches the benchmark’s headline result: an average **Spearman correlation of \(\rho = 0.74\)** across all seven dimensions. For **Omni-GPT-4o**, the reported dimension-wise correlations include **Fluency 0.76**, **Coherence 0.72**, **Faithfulness 0.77**, **Relevance 0.82**, **Aspect Coverage 0.74**, **Sentiment Consistency 0.68**, and **Specificity 0.66**; some are marked significant at **\(p < 0.05\)**.

The paper also reports a user study comparing M-OS summaries against traditional opinion summaries. This study uses **300 participants**, aged **18–50**, with the top-performing generation model **Qwen2.5-72B-Instruct**. Each participant compares **4 pairs of summaries**, anonymized as “Summary 1” and “Summary 2.” The reported result is that participants prefer M-OS summaries **87% of the time on average**. This suggests that metadata-enriched summaries are not only more heavily grounded in product facts but also more useful to end users.

## 6. Limitations, scope, and interpretive significance

M-OS-EVAL is presented as an infrastructural contribution for M-OS, but it is not described as exhaustive [2507.04751]. The benchmark is based on **50 products**, and only **GPT-4o** appears among proprietary evaluator backbones; **Claude-Sonnet 3.5** is explicitly excluded for budget reasons. The work is also effectively **English-centric** and **e-commerce-specific**, and future directions are said to include broader linguistic and cultural coverage, temporal review dynamics, and multimodal content.

Several methodological limits are also explicit. The exact wording of the seven metric definitions is not provided in the supplied appendix text. Prompt-based evaluation may hallucinate or inherit prompt bias, and the authors advise validating prompt reliability before deployment. The rater pool is expert but demographically narrow. More broadly, the benchmark’s scale is sufficient for controlled evaluation but does not fully represent the breadth of real e-commerce catalogs and review ecologies.

Within the supplied literature, an important potential misconception is terminological. The string “MOS” is widely used elsewhere for **Mean Opinion Score**, **Mathematical Optimization Service**, and **Multi-Objective Search**. M-OS-EVAL is unrelated to those usages. It denotes a benchmark for **multi-source opinion summarization**, where “M-OS” means **Multi-Source Opinion Summarization** rather than “MOS” in the audio-quality or optimization sense.

In that sense, M-OS-EVAL’s significance lies less in a new model architecture than in a standardized evaluative frame. It defines what summary quality means for metadata-grounded opinion summarization, supplies expert annotations over a shared candidate pool, and demonstrates that carefully structured LLM evaluators—especially **Omni-Prompt**—can align strongly with expert human judgment. The benchmark therefore functions as both a dataset and an evaluation methodology for a task that had lacked a dedicated standard.

Source: https://www.emergentmind.com/topics/m-os-eval