QF-CES-PROMPT: Comparative Explainable Summarization

• The paper's main contribution is a unified prompt family that generates and evaluates query-focused comparative summaries with an overall Spearman correlation of 0.74.
• QF-CES-PROMPT is embedded in a multi-stage pipeline using Multi-Source Opinion Summarization, reducing generation latency by 40% compared to direct input approaches.
• Automatic evaluators score summaries on clarity, faithfulness, informativeness, format adherence, and query relevance using dimension-specific, reference-free prompts.

QF-CES-PROMPT is a prompt-based framework for Query-Focused Comparative Explainable Summarization (QF-CES) in e-commerce recommendation, introduced in “This Suits You the Best”: Query Focused Comparative Explainable Summarization (Attri et al., 7 Jul 2025). Within that setting, a system receives a user query and the top-3 recommended products, and produces two outputs: a tabular side-by-side comparison and a natural-language final verdict explaining which product best suits the query. QF-CES-PROMPT serves both the generation of such summaries and their automatic evaluation. Its distinctive role is evaluative as much as generative: the framework operationalizes comparative-summary quality with five reference-free dimensions and uses LLMs as judges, achieving an average Spearman correlation of $0.74$ with human judgments for the best evaluator configuration (Attri et al., 7 Jul 2025).

1. Definition and functional scope

QF-CES-PROMPT is defined in the paper as “a set of dimension-dependent prompts” enabling both comparative summary generation and evaluation across five dimensions: clarity, faithfulness, informativeness, format adherence, and query relevance (Attri et al., 7 Jul 2025). It is therefore not a single prompt string, but a prompt family tied to the QF-CES task.

Its placement in the overall framework is dual. On the generation side, it instructs an LLM to produce a query-focused comparative summary in the required structure. On the evaluation side, it instantiates an LLM-as-a-judge, reference-free scoring procedure in which an evaluator model judges a candidate summary directly from the source inputs rather than against a gold reference. The paper explicitly states that it presents “an evaluation of comparative summaries using reference-free metrics with both open- and closed-source LLMs as evaluators” (Attri et al., 7 Jul 2025).

A common misunderstanding is to treat QF-CES-PROMPT as only a generation prompt. In the paper, its more central contribution is arguably the unification of generation control and dimension-specific automatic evaluation under the same task definition. Another common misunderstanding is to interpret the reported $0.74$ as a single-dimension score; the paper states that $0.74$ is the overall average Spearman correlation across the five dimensions for the best evaluator, not a per-dimension correlation (Attri et al., 7 Jul 2025).

2. Position in the QF-CES pipeline

QF-CES-PROMPT is embedded in a multi-stage recommendation-summarization pipeline. The sequence described in the paper is: input query plus top-3 recommended products with metadata and reviews; Multi-Source Opinion Summarization (M-OS) for each product; QF-CES generation from those M-OS representations; and QF-CES evaluation by a second LLM using dimension-specific prompts (Attri et al., 7 Jul 2025).

This placement matters because QF-CES generation is not performed only from raw product data. The paper emphasizes that M-OS acts as an intermediate representation, condensing title, description, key features, specifications, reviews, and average ratings into a single-product summary. The QF-CES generator then operates on these intermediate summaries rather than exclusively on raw evidence. Relative to the Direct Input Approach (DIA), which feeds raw product data directly into comparative generation, the M-OS route reduces generation latency by about 40%, specifically from 16.55 s to 9.99 s per summary in the reported experiment over 50 distinct summaries, each generated 50 times (Attri et al., 7 Jul 2025).

This suggests that QF-CES-PROMPT is best understood not as a generic comparison prompt, but as one component in a broader query-aware, intermediate-summarization pipeline. The prompt’s function is to exploit already distilled evidence and dynamically focus the comparison on attributes salient to the query.

3. Generation-side specification

On the generation side, QF-CES-PROMPT requires a fixed output structure. The generated QF-CES must contain a tabular comparison and a final verdict summary (Attri et al., 7 Jul 2025). The table places products in columns and attributes in rows, and includes titles, prices, ratings, selected attributes, and pros/cons. Missing values are marked as “NA” or “N/A”. The final verdict is a concise natural-language recommendation directly addressing the query.

The paper states that the prompt design comprises three components: a Generation Prompt giving step-by-step instructions for query-relevant tabular comparison and final verdict generation; Evaluation Prompts for each quality dimension with detailed criteria and 1–5 scoring guidelines; and a System Message assigning the LLM a dimension-specific expert role (Attri et al., 7 Jul 2025). However, the exact prompt strings are not published. The paper explicitly does not provide the full generation prompt, the full evaluation prompts, or the exact system messages.

The absence of verbatim prompt text is a defining feature of the framework’s published form. What is available is only the high-level design and required output structure. Accordingly, QF-CES-PROMPT is conceptually specified but not fully reproducible at the prompt-string level from the paper alone.

4. Reference-free evaluation design

The evaluation side of QF-CES-PROMPT is more fully characterized. Each evaluator prompt scores a candidate summary on one of five dimensions using a 1–5 scoring guideline and requires an explanation/justification to improve response quality (Attri et al., 7 Jul 2025). The exact rubric wording for scores 1 through 5 is not provided, but the operational definitions of the dimensions are.

Clarity measures whether the comparative summary is clearly presented, unambiguous, well-structured, readable, and grammatically coherent, including the clarity of the tabular comparison. Faithfulness measures whether the summary is accurate, verifiable, and directly supported by the input data, penalizing unverifiable claims and unsupported generalizations. Informativeness measures coverage of relevant product aspects, including title, base price, final price, dynamically selected attributes, pros, cons, and average rating, with missing values properly marked. Format Adherence measures whether the summary follows the required two-part structure of table plus verdict and whether dynamically selected attributes are properly named rather than placeholders. Query Relevance measures how directly the tabular comparison and final verdict address the user’s query and support an informed decision (Attri et al., 7 Jul 2025).

The evaluator is not reference-based. The paper indicates that, at minimum, the evaluation prompt must receive the user query, the candidate summary, and the input product information or derived evidence needed to assess support and relevance. However, the exact serialized input format is not enumerated (Attri et al., 7 Jul 2025).

To stabilize discrete judge outputs, the paper uses a weighted-expectation scoring procedure: $o = \sum_{k=1}^{j} p(s_k)\times s_k$ where $s_k$ are possible scores and $p(s_k)$ is estimated by repeated sampling, with approximately $n \approx 100$ outputs per input (Attri et al., 7 Jul 2025). At the summary-evaluation level, the paper computes per-query ranking correlations over candidate summaries and averages them across queries, using coefficients such as Spearman’s $\rho$ or Kendall’s $\tau$.

5. Human validation and evaluator performance

The human evaluation benchmark is CES-EVAL, built from 50 instances sampled from MS-Q2P, with 10 model-generated summaries per instance, 5 quality dimensions, and 3 expert annotators, yielding 7,500 ratings in the detailed dataset accounting (Attri et al., 7 Jul 2025). The annotators were one Master’s student, one Pre-Doctoral researcher, and one Doctoral candidate with expertise in opinion summarization; the ethical statement further specifies that all were male, aged 24–32, and had publications or active research in the area.

Human annotation used a 5-point Likert scale and a two-round process. Cases with disagreement of 2 or more points were revisited and discussed. Model identities were hidden, annotators received detailed guidelines, and they were compensated. Inter-annotator agreement improved from Krippendorff’s $\alpha = 0.50$ in Round I to $0.74$0 in Round II (Attri et al., 7 Jul 2025).

Among evaluator LLMs, the paper compares GPT-4o, Llama-3.1-70B, Llama-3.1-8B, Mistral-7B-v0.2, and Mistral-7B-v0.3. The best overall evaluator is Llama-3.1-70B, with dimension-wise Spearman correlations:

• clarity: $0.74$1
• faithfulness: $0.74$2
• informativeness: $0.74$3
• format adherence: $0.74$4
• query relevance: $0.74$5

Their average is $0.74$6 (Attri et al., 7 Jul 2025). GPT-4o performs better on clarity alone, with $0.74$7, but is weaker overall, with an implied average around $0.74$8. This is one of the paper’s main empirical findings: an open-source evaluator can outperform a proprietary one on this task.

6. Data, reproducibility, and limitations

The framework is evaluated in the context of MS-Q2P, described in the abstract as comprising 7,500 queries mapped to 22,500 recommended products, while a later dataset-scale summary reports about 7,752 queries and 23,256 products; the discrepancy is present in the source text itself (Attri et al., 7 Jul 2025). The human-evaluation subset is CES-EVAL.

Reproducibility is only partial. The paper reports decoding settings: for open-source summary generation, top_k = 25, top_p = 0.95, number of beams = 3, and temperature = 0.2; for OpenAI generation, temperature = 0; for open-source evaluation, n = 100 and temperature = 0.2; for OpenAI evaluation, temperature = 0. Experiments were run on 8 NVIDIA A100-SXM4-80GB GPUs (Attri et al., 7 Jul 2025). These details make the evaluation regime reproducible at the systems level, but not at the prompt-template level.

The paper is explicit about several limitations. QF-CES-PROMPT is task-specific: “Its broader applicability requires further study and potential prompt adjustments.” The framework may produce hallucinations, especially in complex cases. The source product dataset averages only 10 reviews per product, which may limit downstream faithfulness and informativeness. During generation, LLMs sometimes struggled with products having extensive specifications, yielding incomplete or stalled summaries. The evaluator comparison is limited to a small set of models, with some proprietary systems omitted for cost reasons. The human annotator pool is expert but demographically narrow (Attri et al., 7 Jul 2025).

A final misconception addressed by the paper is that QF-CES-PROMPT is a fully specified benchmark artifact. It is not. The contribution is conceptually and empirically clear, but the paper explicitly omits the full exact prompt text, exact 1–5 rubric wording, explicit output schema, and example evaluator outputs. Its importance therefore lies less in prompt-string publication than in establishing a task-specific, dimension-grounded, reference-free evaluation framework for comparative explainable summarization.