PeReGrINE: Evaluating Personalized Review Fidelity with User Item Graph Context

Abstract: We introduce PeReGrINE, a benchmark and evaluation framework for personalized review generation grounded in graph-structured user--item evidence. PeReGrINE restructures Amazon Reviews 2023 into a temporally consistent bipartite graph, where each target review is conditioned on bounded evidence from user history, item context, and neighborhood interactions under explicit temporal cutoffs. To represent persistent user preferences without conditioning directly on sparse raw histories, we compute a User Style Parameter that summarizes each user's linguistic and affective tendencies over prior reviews. This setup supports controlled comparison of four graph-derived retrieval settings: product-only, user-only, neighbor-only, and combined evidence. Beyond standard generation metrics, we introduce Dissonance Analysis, a macro-level evaluation framework that measures deviation from expected user style and product-level consensus. We also study visual evidence as an auxiliary context source and find that it can improve textual quality in some settings, while graph-derived evidence remains the main driver of personalization and consistency. Across product categories, PeReGrINE offers a reproducible way to study how evidence composition affects review fidelity, personalization, and grounding in retrieval-conditioned LLMs.

• The paper introduces the PeReGrINE framework, a benchmark for personalized review generation using graph-based user-item context and multimodal evidence.
• The paper details a controlled retrieval paradigm with hard temporal constraints and distinct evidence paths from user history, item metadata, and visual cues.
• The paper demonstrates that combined evidence yields balanced stylistic personalization and product grounding, outperforming baselines in ROUGE, BERTScore, and dissonance metrics.

PeReGrINE: Evaluating Personalized Review Fidelity with User–Item Graph Context

Introduction and Motivation

The "PeReGrINE" framework presents a controlled, interpretable benchmark for personalized review generation anchored in graph-based user-item context and augmented by multimodal evidence (2604.07788). The motivation stems from the dual challenge of synthesizing review text that is simultaneously grounded in product details and faithful to a reviewer’s individual style—a scenario not adequately captured by existing methodologies or evaluation protocols. PeReGrINE restructures the Amazon Reviews 2023 data into temporally-filtered bipartite graphs and systematically disentangles user history, item metadata/neighborhood, and auxiliary modalities (e.g., catalog and review images) as conditional evidence sources.

System Overview

PeReGrINE’s pipeline operationalizes a three-stage approach: (1) summarizing a user's persistent stylistic tendencies via a computed User Style Parameter, (2) retrieving item neighborhood and user history reviews under strict temporal cutoffs, and (3) conditioning a LLM on this evidence for review synthesis.

Figure 1: The system computes a user style summary, retrieves item-side and user-side evidence from the graph, and then conditions a LLM on the resulting context.

Central to PeReGrINE’s design is the product-anchored retrieval paradigm. Unlike prior frameworks that risk information leakage by seeding retrieval with fragments of the gold review, PeReGrINE strictly anchors all item-side evidence on pre-existing product metadata and pre-target-timestamp neighbor reviews. User history is abstracted either as a summary vector or selected representative prior reviews, promoting stylistic personalization while maintaining computational tractability.

Retrieval Evidence Paradigms

The benchmark explicitly supports combinatorial evidence regimes: product-only, user-only, neighbor-only (other users’ reviews of the item), and combined settings. This bipartite organization allows precise attribution of grounding (item fidelity) versus personalization (reviewer voice) and exposes the contribution of each evidence pathway.

Figure 3: User-history retrieval corresponds to the red edges, item-neighbor retrieval corresponds to the green edges, and the combined setting uses both.

The structure establishes a methodological baseline for controlled ablation across evidence settings, with temporal filtering ensuring no training-time or evaluation-time leakage.

Methodology and Reproducibility

User Style Parameter

PeReGrINE defines the User Style Parameter as an 11-D feature vector summarizing stylistic, sentiment, and lexical characteristics averaged over eligible user history. This proxy encodes stable behavioral traits (length, sentiment, punctuation, persona cues) while sidestepping the sparsity and computational complexity of raw full-history conditioning.

Temporal and Evidence Constraints

All retrieval functions enforce hard temporal cutoffs, i.e., only interactions prior to the generation target are valid. This constraint is critical for isolating the causal role of evidence and supporting fair comparison with real-world deployment constraints.

Multimodal Context Integration

The primary novelty in evidence construction is that visual signals (catalog and review images) are translated into captions and appended to the generative prompt after ranking, not during candidate retrieval. While this approach simplifies benchmarking and prevents spurious visual anchoring, it does not allow direct study of truly multimodal retrieval—an explicit limitation and avenue for future work.

Evaluation Protocols and Metrics

The evaluation explicitly distinguishes between micro-level metrics (ROUGE-L, BERTScore, METEOR for text/title, rating accuracy/MAE/RMSE) and macro-level "Dissonance Analysis," which quantifies the deviation of the generated review from (1) expected user style, (2) product consensus clusters, and (3) sentiment-label congruence.

The Dissonance framework provides interpretable heuristics for quantifying behavioral drift—crucial in evaluating personalized generation tasks that can otherwise overfit to general user or product archetypes.

Figure 4: (Left) Per-metric mean differences between the multimodal combined setting and a text-only combined setting (right) Category-wise performance for the combined multimodal setting. Images improve text alignment and product grounding, but gains on rating fidelity are limited.

Experimental Findings

Evidence Source Ablation

The systematic ablations reveal that:

• Product-only evidence optimizes grounding and title-text consistency, with lowest product dissonance, but yields generic style.
• User-only evidence minimizes user dissonance and maximizes rating prediction accuracy at the expense of product fidelity.
• Combined evidence provides the most balanced tradeoff, yielding the highest overall ROUGE/BERTScore and lowest aggregate dissonance.

Baseline Comparisons

In direct, protocol-matched comparisons, PeReGrINE surpasses both LaMP and PGraphRAG on review-text alignment metrics (e.g., ROUGE-L, BERTScore-F1), while LaMP remains stronger for title-oriented metrics and PGraphRAG achieves lower rating error. The joint generation variant (single-review for all subtasks) underperforms dramatically on title and rating, demonstrating the importance of specialized prompting and evidence selection for each subtask.

Multimodal Augmentation

Augmenting with visual evidence gives marginal improvements in text overlap and product dissonance, but these gains are secondary to those achieved by quality graph-based retrieval:

• Visual evidence primarily enhances factual grounding but does not substantially affect stylization or rating accuracy.
• The effect of image data is highly category-dependent, being most pronounced in domains with high catalog image coverage.
• The retrieval ranker itself remaining text-centric is a limiting factor; images only inform the final prompt context.

Cross-Domain Analysis

Strong category effects are evident: “All Beauty” is an easier domain for generation alignment, while categories like “Sports” and “Toys & Games” show more narrative variability and weaker text metric alignment, emphasizing the importance of domain-driven evaluation. Product consensus and rating predictability also vary substantially by domain.

Figure 2: Illustration of input structure; the user profile shapes stylistic evidence, while product information dictates grounding context.

Theoretical and Practical Implications

PeReGrINE advances the evaluation and benchmarking of personalized generation by:

• Operationalizing strict temporal and protocol constraints to mimic real-world deployment (no answer leakage).
• Providing a transparent, replicable framework that disentangles the roles of user- and item-derived evidence.
• Demonstrating that visual augmentation is auxiliary to robust text-based graph retrieval for both grounding and personalization tasks.

Practically, these results imply that for current retrieval-augmented LLMs, graph-derived context remains the principal driver for behavioral fidelity, while multimodal evidence can augment but not substitute the core retrieval mechanisms.

Limitations and Future Directions

Two main constraints are explicit: PeReGrINE does not conduct truly multimodal retrieval at ranking time, and the User Style Parameter is an interpretable but coarse behavioral proxy. Future work should:

• Integrate image and potentially video embeddings directly into the retrieval ranking stage to test joint vision-language evidence selection.
• Broaden product-side context to richer modalities (videos, frame-level cues) and perform feature-group ablations on both style and dissonance metrics for interpretability.
• Expand to handle sparser graph neighborhoods (cold start) and address long-tail behavioral modeling.
• Introduce fine-grained ablation analysis for each component of the stylometric and dissonance metrics.

Conclusion

PeReGrINE provides a comprehensive, methodologically robust benchmark for evaluating personalized review generation conditioned on explicit, temporally-vetted user-item graphs and auxiliary visual context. The framework demonstrates, in well-controlled ablations and cross-model comparisons, that graph-based retrieval dominates grounding and personalization, while visual evidence acts as a useful secondary cue rather than the main driver. The design and findings set a high empirical standard for future personalized, multimodal, and retrieval-augmented text generation benchmarks and practical systems.