M-OS-EVAL: Benchmark for Multi-Source Opinion Summaries

• M-OS-EVAL is a benchmark that assesses metadata-grounded opinion summaries by integrating subjective reviews with objective product details.
• It applies a comprehensive evaluation protocol across seven quality dimensions, showing improved inter-rater agreement through a two-round expert annotation process.
• The dataset supports prompt-based LLM evaluators, with experiments demonstrating high performance from models like Qwen2.5-72B-Instruct and GPT-4o.

Searching arXiv for papers directly relevant to “M-OS-EVAL” and adjacent evaluation methodology in multi-source opinion summarization. M-OS-EVAL is a reference-free benchmark dataset for evaluating multi-source opinion summaries in e-commerce settings. It was introduced together with Multi-Source Opinion Summarization (M-OS), a task that extends review-only opinion summarization by incorporating product metadata such as title, description, key features, specifications, and average ratings alongside customer reviews. Its central purpose is to provide a benchmark tailored to summaries that must jointly capture subjective opinions and objective product attributes, and to support the development of LLM-based evaluators that correlate with expert human judgment across seven quality dimensions: fluency, coherence, relevance, faithfulness, aspect coverage, sentiment consistency, and specificity (Attri et al., 7 Jul 2025).

1. Definition within multi-source opinion summarization

M-OS-EVAL is defined in the context of M-OS rather than traditional opinion summarization. In the underlying task, the input is not limited to customer reviews; it includes both subjective evidence from reviews and objective product metadata from titles, descriptions, key features, specifications, and ratings. The benchmark therefore targets summaries that integrate heterogeneous sources into a single output.

This task formulation addresses a specific limitation of review-only summarization: a summary may capture consensus sentiment while omitting concrete product attributes needed for product comparison and purchase decisions. M-OS-EVAL operationalizes evaluation for this richer setting by treating metadata-grounded summarization as a distinct problem rather than as a minor variant of conventional opinion summarization. A plausible implication is that evaluation criteria for M-OS must reward not only linguistic quality and sentiment fidelity but also correct integration of factual product information.

The benchmark is also motivated by the inadequacy of standard reference-based automatic metrics such as ROUGE, BLEU, BERTSCORE, and BARTSCORE for this setting. The benchmark treats M-OS evaluation as fundamentally reference-free, because metadata-enriched summaries may be factually richer and structurally different from any single reference while still being superior.

2. Dataset construction and annotation protocol

M-OS-EVAL is built on top of M-OS-DATA, a proprietary product-metadata dataset created through collaboration between the authors’ university lab and a major e-commerce company (Attri et al., 7 Jul 2025). M-OS-DATA contains 7,752 unique queries and 23,256 products, with each product including title, description, features, specifications, reviews, and average rating. The reported averages are 10 reviews per product, 242.6 words of specifications, 17.99 words of reviews, 105.79 words of description, and 24.64 words of key features.

The benchmark itself is constructed from the M-OS-DATA test set. It uses 50 products, each paired with 14 model-generated summaries. Every summary is evaluated on 7 dimensions, and each summary-dimension pair is rated by 3 expert raters. This yields 4,900 summary-dimension instances and 14,700 total ratings:

$$3 \text{ raters} \times 50 \text{ products} \times 14 \text{ summaries} \times 7 \text{ dimensions} = 14{,}700.$$

The 14 summaries per product come from 14 LLMs: Mistral-7B-Instruct-v0.3, Meta-Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.2, gemma-7b-it, vicuna-7b-v1.5, zephyr-7b-beta, GPT-4o, Gemma-2-9b-it, Mistral-Small-Instruct-2409, Mixtral-8x7B-Instruct-v0.1, Qwen2.5-7B-Instruct, Qwen2.5-32B-Instruct, Qwen2.5-72B-Instruct, and Meta-Llama-3.1-70B-Instruct.

The human annotation protocol uses 3 experienced raters: one Master’s-level, one Pre-Doctoral, and one Doctoral-level. All were male, aged 24–32, and had publications or active research in opinion summarization or were working in the domain. Annotation proceeded in two rounds. In Round I, ratings were assigned independently. In Round II, if ratings differed by 2 or more points, raters re-evaluated through discussion until the discrepancy was reduced to 1 point or less. Model identities were undisclosed to reduce bias.

Component	Value	Notes
Products	50	From the M-OS-DATA test set
Summaries per product	14	One from each evaluated LLM
Dimensions	7	Human and LLM-evaluator targets
Raters per item	3	Expert raters
Summary-dimension instances	4,900	$50 \times 14 \times 7$
Total ratings	14,700	$3 \times 50 \times 14 \times 7$

3. Evaluation dimensions and agreement structure

M-OS-EVAL evaluates summaries along seven dimensions (Attri et al., 7 Jul 2025). The paper points to an appendix for exact definitions, but the main text characterizes the intended focus of each dimension clearly enough to support practical use.

Dimension	Evaluative focus
Fluency	Grammaticality, readability, and naturalness
Coherence	Logical flow and organization
Relevance	Inclusion of important product information
Faithfulness	Factual correctness relative to sources
Aspect Coverage	Coverage of important product aspects
Sentiment Consistency	Alignment with review sentiment
Specificity	Product-specific detail and precision

Each dimension is scored on a 5-point Likert scale. The annotation analysis reports Krippendorff’s alpha for both annotation rounds. Overall agreement rises from 0.70 in Round I to 0.86 in Round II. Dimension-wise values are: Fluency 0.73 → 0.88, Coherence 0.67 → 0.82, Relevance 0.69 → 0.85, Faithfulness 0.79 → 0.91, Aspect Coverage 0.77 → 0.89, Sentiment Consistency 0.66 → 0.86, and Specificity 0.61 → 0.84.

These values are interpreted in the benchmark using the reported ranges 0.61–0.80 = moderate and 0.81–1.00 = substantial. The increase from Round I to Round II indicates that the combination of expert raters, detailed guidelines, and discussion-based reconciliation materially improves consistency. A plausible implication is that dimensions tied to product metadata, especially faithfulness and aspect coverage, benefit from more explicit rubric standardization than dimensions driven by narrative judgment.

4. Prompt-based LLM evaluators

M-OS-EVAL is not only a human-annotated benchmark; it is also a testbed for LLM-based evaluation prompts. The paper introduces M-OS-PROMPTS for both generation and evaluation, with the evaluation side centered on M-OS-EVAL-PROMPTS (Attri et al., 7 Jul 2025).

Each evaluation prompt contains four components: a System Message, a Task Description, Evaluation Criteria, and an Evaluation Step. Two prompt families are defined. Omni-Prompt is metric-independent and uses a common structure across dimensions with a dynamic metric slot. Spectra-Prompts are metric-dependent, with one specialized prompt per dimension. These are compared against prior prompt-based evaluators from Siledar et al.: Op-I-Prompt and Op-Prompts.

The evaluator backbones include Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.2, Mistral-7B-Instruct-v0.3, Llama-3.1-70B-Instruct, and GPT-4o. Scoring follows the weighted-average scheme associated with G-Eval:

$o = \sum_{k=1}^{j} p(s_k)\times s_k$

where $s_k$ are possible scores and $p(s_k)$ are their LLM-determined probabilities. The paper states that $p(s_k)$ is estimated by sampling with $n \approx 100$ outputs per input.

Meta-evaluation uses summary-level correlation following Bhandari et al., with Spearman $(\rho)$ and Kendall Tau $(\tau)$ as the reported correlation measures. The benchmark therefore serves two roles simultaneously: it is a dataset of expert judgments over M-OS summaries, and it is an evaluation target for prompt-engineered LLM judges.

5. Reported empirical findings

The benchmark supports two main experimental lines: evaluation of M-OS generation systems and evaluation of LLM judges (Attri et al., 7 Jul 2025).

For human evaluation of generated summaries, the best overall model is Qwen2.5-72B-Instruct with an average score of 4.186, followed by GPT-4o at 4.169 and Qwen2.5-32B-Instruct at 4.149. The reported pattern is that most models score highly on fluency, while larger differences appear on specificity. Open-source models are described as competitive with GPT-4o, and Qwen2.5-72B-Instruct is reported as outperforming GPT-4o overall.

For meta-evaluation, Spectra-GPT-4o achieves the best average Spearman correlation of 0.70 among metric-dependent prompts. In the metric-independent setting, Omni-Prompt + GPT-4o reaches the benchmark’s headline result: an average Spearman correlation of $50 \times 14 \times 7$0 across all seven dimensions. For Omni-GPT-4o, the reported dimension-wise correlations include Fluency 0.76, Coherence 0.72, Faithfulness 0.77, Relevance 0.82, Aspect Coverage 0.74, Sentiment Consistency 0.68, and Specificity 0.66; some are marked significant at $50 \times 14 \times 7$1.

The paper also reports a user study comparing M-OS summaries against traditional opinion summaries. This study uses 300 participants, aged 18–50, with the top-performing generation model Qwen2.5-72B-Instruct. Each participant compares 4 pairs of summaries, anonymized as “Summary 1” and “Summary 2.” The reported result is that participants prefer M-OS summaries 87% of the time on average. This suggests that metadata-enriched summaries are not only more heavily grounded in product facts but also more useful to end users.

6. Limitations, scope, and interpretive significance

M-OS-EVAL is presented as an infrastructural contribution for M-OS, but it is not described as exhaustive (Attri et al., 7 Jul 2025). The benchmark is based on 50 products, and only GPT-4o appears among proprietary evaluator backbones; Claude-Sonnet 3.5 is explicitly excluded for budget reasons. The work is also effectively English-centric and e-commerce-specific, and future directions are said to include broader linguistic and cultural coverage, temporal review dynamics, and multimodal content.

Several methodological limits are also explicit. The exact wording of the seven metric definitions is not provided in the supplied appendix text. Prompt-based evaluation may hallucinate or inherit prompt bias, and the authors advise validating prompt reliability before deployment. The rater pool is expert but demographically narrow. More broadly, the benchmark’s scale is sufficient for controlled evaluation but does not fully represent the breadth of real e-commerce catalogs and review ecologies.

Within the supplied literature, an important potential misconception is terminological. The string “MOS” is widely used elsewhere for Mean Opinion Score, Mathematical Optimization Service, and Multi-Objective Search. M-OS-EVAL is unrelated to those usages. It denotes a benchmark for multi-source opinion summarization, where “M-OS” means Multi-Source Opinion Summarization rather than “MOS” in the audio-quality or optimization sense.

In that sense, M-OS-EVAL’s significance lies less in a new model architecture than in a standardized evaluative frame. It defines what summary quality means for metadata-grounded opinion summarization, supplies expert annotations over a shared candidate pool, and demonstrates that carefully structured LLM evaluators—especially Omni-Prompt—can align strongly with expert human judgment. The benchmark therefore functions as both a dataset and an evaluation methodology for a task that had lacked a dedicated standard.