---
title: 'QF-CES-PROMPT: Comparative Explainable Summarization'
url: https://www.emergentmind.com/topics/qf-ces-prompt
type: topic
---

# QF-CES-PROMPT: Comparative Explainable Summarization

QF-CES-PROMPT is a prompt-based framework for **Query-Focused Comparative Explainable Summarization (QF-CES)** in e-commerce recommendation, introduced in “This Suits You the Best”: Query Focused Comparative Explainable Summarization [2507.04733]. Within that setting, a system receives a user query and the top-3 recommended products, and produces two outputs: a **tabular side-by-side comparison** and a **natural-language final verdict** explaining which product best suits the query. QF-CES-PROMPT serves both the **generation** of such summaries and their **automatic evaluation**. Its distinctive role is evaluative as much as generative: the framework operationalizes comparative-summary quality with five reference-free dimensions and uses LLMs as judges, achieving an average Spearman correlation of \(0.74\) with human judgments for the best evaluator configuration [2507.04733].

## 1. Definition and functional scope

QF-CES-PROMPT is defined in the paper as “a set of dimension-dependent prompts” enabling both comparative summary generation and evaluation across five dimensions: **clarity**, **faithfulness**, **informativeness**, **format adherence**, and **query relevance** [2507.04733]. It is therefore not a single prompt string, but a prompt family tied to the QF-CES task.

Its placement in the overall framework is dual. On the generation side, it instructs an LLM to produce a query-focused comparative summary in the required structure. On the evaluation side, it instantiates an **LLM-as-a-judge**, reference-free scoring procedure in which an evaluator model judges a candidate summary directly from the source inputs rather than against a gold reference. The paper explicitly states that it presents “an evaluation of comparative summaries using reference-free metrics with both open- and closed-source LLMs as evaluators” [2507.04733].

A common misunderstanding is to treat QF-CES-PROMPT as only a generation prompt. In the paper, its more central contribution is arguably the unification of **generation control** and **dimension-specific automatic evaluation** under the same task definition. Another common misunderstanding is to interpret the reported \(0.74\) as a single-dimension score; the paper states that \(0.74\) is the **overall average Spearman correlation across the five dimensions** for the best evaluator, not a per-dimension correlation [2507.04733].

## 2. Position in the QF-CES pipeline

QF-CES-PROMPT is embedded in a multi-stage recommendation-summarization pipeline. The sequence described in the paper is: input query plus top-3 recommended products with metadata and reviews; **Multi-Source Opinion Summarization (M-OS)** for each product; QF-CES generation from those M-OS representations; and QF-CES evaluation by a second LLM using dimension-specific prompts [2507.04733].

This placement matters because QF-CES generation is not performed only from raw product data. The paper emphasizes that **M-OS acts as an intermediate representation**, condensing title, description, key features, specifications, reviews, and average ratings into a single-product summary. The QF-CES generator then operates on these intermediate summaries rather than exclusively on raw evidence. Relative to the **Direct Input Approach (DIA)**, which feeds raw product data directly into comparative generation, the M-OS route reduces generation latency by about **40%**, specifically from **16.55 s** to **9.99 s** per summary in the reported experiment over 50 distinct summaries, each generated 50 times [2507.04733].

This suggests that QF-CES-PROMPT is best understood not as a generic comparison prompt, but as one component in a broader **query-aware, intermediate-summarization pipeline**. The prompt’s function is to exploit already distilled evidence and dynamically focus the comparison on attributes salient to the query.

## 3. Generation-side specification

On the generation side, QF-CES-PROMPT requires a fixed output structure. The generated QF-CES must contain a **tabular comparison** and a **final verdict summary** [2507.04733]. The table places products in columns and attributes in rows, and includes titles, prices, ratings, selected attributes, and pros/cons. Missing values are marked as **“NA”** or **“N/A”**. The final verdict is a concise natural-language recommendation directly addressing the query.

The paper states that the prompt design comprises three components: a **Generation Prompt** giving step-by-step instructions for query-relevant tabular comparison and final verdict generation; **Evaluation Prompts** for each quality dimension with detailed criteria and 1–5 scoring guidelines; and a **System Message** assigning the LLM a dimension-specific expert role [2507.04733]. However, the exact prompt strings are not published. The paper explicitly does **not** provide the full generation prompt, the full evaluation prompts, or the exact system messages.

The absence of verbatim prompt text is a defining feature of the framework’s published form. What is available is only the high-level design and required output structure. Accordingly, QF-CES-PROMPT is conceptually specified but not fully reproducible at the prompt-string level from the paper alone.

## 4. Reference-free evaluation design

The evaluation side of QF-CES-PROMPT is more fully characterized. Each evaluator prompt scores a candidate summary on one of five dimensions using a **1–5 scoring guideline** and requires an **explanation/justification** to improve response quality [2507.04733]. The exact rubric wording for scores 1 through 5 is not provided, but the operational definitions of the dimensions are.

**Clarity** measures whether the comparative summary is clearly presented, unambiguous, well-structured, readable, and grammatically coherent, including the clarity of the tabular comparison. **Faithfulness** measures whether the summary is accurate, verifiable, and directly supported by the input data, penalizing unverifiable claims and unsupported generalizations. **Informativeness** measures coverage of relevant product aspects, including title, base price, final price, dynamically selected attributes, pros, cons, and average rating, with missing values properly marked. **Format Adherence** measures whether the summary follows the required two-part structure of table plus verdict and whether dynamically selected attributes are properly named rather than placeholders. **Query Relevance** measures how directly the tabular comparison and final verdict address the user’s query and support an informed decision [2507.04733].

The evaluator is not reference-based. The paper indicates that, at minimum, the evaluation prompt must receive the **user query**, the **candidate summary**, and the **input product information or derived evidence** needed to assess support and relevance. However, the exact serialized input format is not enumerated [2507.04733].

To stabilize discrete judge outputs, the paper uses a weighted-expectation scoring procedure:
\[
o = \sum_{k=1}^{j} p(s_k)\times s_k
\]
where \(s_k\) are possible scores and \(p(s_k)\) is estimated by repeated sampling, with approximately \(n \approx 100\) outputs per input [2507.04733]. At the summary-evaluation level, the paper computes per-query ranking correlations over candidate summaries and averages them across queries, using coefficients such as Spearman’s \(\rho\) or Kendall’s \(\tau\).

## 5. Human validation and evaluator performance

The human evaluation benchmark is **CES-EVAL**, built from **50** instances sampled from MS-Q2P, with **10** model-generated summaries per instance, **5** quality dimensions, and **3** expert annotators, yielding **7,500 ratings** in the detailed dataset accounting [2507.04733]. The annotators were one Master’s student, one Pre-Doctoral researcher, and one Doctoral candidate with expertise in opinion summarization; the ethical statement further specifies that all were male, aged 24–32, and had publications or active research in the area.

Human annotation used a **5-point Likert scale** and a **two-round process**. Cases with disagreement of **2 or more points** were revisited and discussed. Model identities were hidden, annotators received detailed guidelines, and they were compensated. Inter-annotator agreement improved from **Krippendorff’s \(\alpha = 0.50\)** in Round I to **\(\alpha = 0.80\)** in Round II [2507.04733].

Among evaluator LLMs, the paper compares **GPT-4o**, **Llama-3.1-70B**, **Llama-3.1-8B**, **Mistral-7B-v0.2**, and **Mistral-7B-v0.3**. The best overall evaluator is **Llama-3.1-70B**, with dimension-wise Spearman correlations:
- clarity: \(0.70\)
- faithfulness: \(0.77\)
- informativeness: \(0.82\)
- format adherence: \(0.73\)
- query relevance: \(0.68\)

Their average is \(0.74\) [2507.04733]. GPT-4o performs better on clarity alone, with \(0.77\), but is weaker overall, with an implied average around \(0.71\). This is one of the paper’s main empirical findings: an open-source evaluator can outperform a proprietary one on this task.

## 6. Data, reproducibility, and limitations

The framework is evaluated in the context of **MS-Q2P**, described in the abstract as comprising **7,500 queries** mapped to **22,500 recommended products**, while a later dataset-scale summary reports about **7,752 queries** and **23,256 products**; the discrepancy is present in the source text itself [2507.04733]. The human-evaluation subset is CES-EVAL.

Reproducibility is only partial. The paper reports decoding settings: for open-source summary generation, `top_k = 25`, `top_p = 0.95`, `number of beams = 3`, and `temperature = 0.2`; for OpenAI generation, `temperature = 0`; for open-source evaluation, `n = 100` and `temperature = 0.2`; for OpenAI evaluation, `temperature = 0`. Experiments were run on **8 NVIDIA A100-SXM4-80GB GPUs** [2507.04733]. These details make the evaluation regime reproducible at the systems level, but not at the prompt-template level.

The paper is explicit about several limitations. QF-CES-PROMPT is **task-specific**: “Its broader applicability requires further study and potential prompt adjustments.” The framework may produce **hallucinations**, especially in complex cases. The source product dataset averages only **10 reviews per product**, which may limit downstream faithfulness and informativeness. During generation, LLMs sometimes struggled with products having extensive specifications, yielding incomplete or stalled summaries. The evaluator comparison is limited to a small set of models, with some proprietary systems omitted for cost reasons. The human annotator pool is expert but demographically narrow [2507.04733].

A final misconception addressed by the paper is that QF-CES-PROMPT is a fully specified benchmark artifact. It is not. The contribution is conceptually and empirically clear, but the paper explicitly omits the **full exact prompt text**, **exact 1–5 rubric wording**, **explicit output schema**, and **example evaluator outputs**. Its importance therefore lies less in prompt-string publication than in establishing a **task-specific, dimension-grounded, reference-free evaluation framework** for comparative explainable summarization.

Source: https://www.emergentmind.com/topics/qf-ces-prompt