CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors

Abstract: While LLMs have demonstrated remarkable potential in Question Answering (QA), evaluating personalization remains a critical bottleneck. Existing paradigms predominantly rely on lexical-level similarity or manual heuristics, often lacking sufficient data-driven validation. We address this by mining Community-Individual Preference Divergence (CIPD), where individual choices override consensus, to distill six key personalization factors as evaluative dimensions. Accordingly, we introduce CoPA, a benchmark with 1,985 user profiles for fine-grained, factor-level assessment. By quantifying the alignment between model outputs and user-specific cognitive preferences inferred from interaction patterns, CoPA provides a more comprehensive and discriminative standard for evaluating personalized QA than generic metrics. The code is available at https://github.com/bjzgcai/CoPA.

• The paper introduces a novel data-driven pipeline that extracts cognitive factors from user behavior to benchmark personalized QA models.
• It operationalizes Community-Individual Preference Divergence (CIPD) and leverages a multi-stage factor distillation process to quantify personalization using six distinct cognitive metrics.
• Experimental results demonstrate that profile-based personalization outperforms traditional methods, with high alignment (Spearman ρ ≈ 0.75) and strong domain generalization.

CoPA: Benchmarking Personalized QA with Data-Informed Cognitive Factors

Introduction and Motivation

Despite the increasing deployment of LLMs in personalized question answering (QA), the evaluation of genuine personalization is undertheorized, with most current evaluation paradigms anchored in lexical overlap metrics or rule-based heuristics that lack cognitive grounding. The paper "CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors" (2604.14773) addresses this critical gap by proposing a systematic, data-driven pipeline for extracting personalization criteria from real-world user behavior and deploying these as evaluative axes in a new benchmarking suite.

The methodological core is the identification and formalization of Community-Individual Preference Divergence (CIPD), operationalized in scenarios where individual user choices systematically diverge from the StackExchange community consensus. By mining CIPD phenomena, the study uncovers underlying cognitive factors that inform personalization at scale, providing a pipeline that is inherently more discriminative than previous heuristics.

CIPD Phenomenon: Empirical Characterization

The StackExchange platform provides a massive testbed for modeling CIPD, containing diverse QA pairs across engineering, science, lifestyle, and leisure domains, each with explicit indicators of individual (accepted) and community (top-voted) solution preference.

A canonical example is presented where a user accepts an empirical, code-based solution over the community’s schematic favorite:

Figure 1: Example of Community-Individual Preference Divergence (CIPD) from Stack Overflow, illustrating differential valuation of empirical versus schematic solutions.

Quantitative analyses reveal the prevalence and variability of CIPD across knowledge domains:

Figure 2: CIPD question proportions by category, indicating CIPD is not restricted to subjective or open-ended domains.

The rank and score differential statistics further demonstrate that accepted answers reflect idiosyncratic, not merely temporal or exposure-related, user priorities:

Figure 3: Distribution of CIPD question ranks; substantial cases show users deviate from community consensus despite visibility bias toward top-voted answers.

Factor Distillation: From Behavioral Explanations to Cognitive Dimensions

To uncover the latent factors that compute user personalization, the paper details a multi-stage factor distillation procedure (Figure 4). High-quality CIPD examples are paired with user QA history for context. LLMs are then prompted to generate structured rationales for user acceptance decisions, leveraging schema-level (reason, theory, explanation) outputs, which are aggregated, clustered, and distilled into core factors via multi-model consensus and expert validation:

Figure 4: Schematic pipeline for factor distillation from raw CIPD cases to stable cognitive dimensions.

The six distilled cognitive factors forming the CoPA evaluative schema are:

• Cognitive Trust (CT): Domain-aligned epistemic credibility and verifiability requirements.
• Situational Anchoring (SA): Calibration to immediate context and scenario dependencies.
• Schema Consistency (SC): Coherence with prior knowledge and internal user models.
• Cognitive Load Management (CLM): Adaptation of complexity to the user’s cognitive capacity.
• Metacognitive Scaffolding (MS): Provision of structural cues for critical reflection and higher-order reasoning.
• Affective and Motivational Resonance (AMR): Emotional and motivational consonance with user state.

CoPA Benchmark Construction

The CoPA benchmark consists of 1,985 user profiles sampled from the top-quality segment of StackExchange, with each profile parameterized by the six factors above. For each question-answer instance, model outputs are rated for alignment with user-specific cognitive profiles using a 3-point Likert scale per factor, establishing a fine-grained measure of personalization.

The response evaluation pipeline quantifies degree of personalization through direct comparison of model answers with individualized factor preferences:

Figure 5: Evaluation pipeline for scoring model responses against six-factor user profiles.

Experimental Results

Factor-Efficacy and Necessity

CoPA’s factor-based scoring robustly discriminates personalized decisions, achieving substantially higher accuracy and lower tie rates than LLM-as-a-Judge, Chain-of-Thought, and purely heuristic metrics. Ablation studies confirm the non-redundancy of each factor: removal of any dimension degrades performance across all accuracy and margin metrics.

Factor Correlation Structure

Inter-factor Spearman correlation analysis validates that while certain axes (e.g., Cognitive Trust and Schema Consistency) are moderately correlated, no pair exhibits excessive redundancy (all $r < 0.9$), with Cognitive Load Management especially orthogonal, confirming comprehensive (rather than collinear) factor coverage.

Figure 6: Pairwise Spearman correlation matrix for the six cognitive factors, highlighting relative independence and discriminant validity.

Benchmarking Personalization Models

A spectrum of personalization strategies—no profile, time-based, RAG-based, and compressed profile—were tested with SOTA LLMs (Qwen3-8B and GPT-4o-mini). Profile-Personalization outperforms all other methods, especially in domains requiring long-range context modeling. Inclusion of explicit, compressed profiles yields responses with demonstrably higher alignment to user preferences, as measured by CoPA.

Human-LLM scoring alignment is high (Spearman $\rho$ ≈ 0.75), supporting CoPA’s validity as a practical evaluation framework.

Domain Generalization and Ablation

Factor-driven personalization retains robust gains when ported to external datasets (UPGC-QA and LaMP-QA), confirming the domain-transferability of the cognitive dimensions mined in the source StackExchange corpus.

Theoretical Implications

The results support the thesis that explicit cognitive factor profiling, grounded in empirical behavioral divergences like CIPD, provides a finer, psychologically valid axis along which to measure and engineer personalization in LLM QA agents. Unlike traditional approaches focused on surface textual matching, CoPA operationalizes personalization as the alignment of generative outputs to rich psychometric attributes inferred from context and trajectory, facilitating controlled benchmarking and model development across a spectrum of real-world application domains.

Practical Implications and Future Directions

The introduction of the CoPA benchmark and factor distillation pipeline enables statistically valid, interpretable factor-level diagnostics of LLM-based personalized QA, with direct applications in user-facing knowledge delivery, education, and domain-specific assistant engineering. The findings also highlight that optimizing for factors such as Metacognitive Scaffolding remains nontrivial and open, identifying a frontier for architectural and prompting advances.

Figure 7: Relationship between historical context length (K) and model personalization performance, highlighting trade-offs between response fidelity and inference cost.

Conclusion

By algorithmically mining Community-Individual Preference Divergence and distilling behavioral rationales into a controlled cognitive factor framework, "CoPA" (2604.14773) advances evaluation methodology for personalized QA well beyond lexical proxies and rules-of-thumb. The robust, empirically grounded factor taxonomy enables discriminative, interpretable, and generalizable assessment of model-user alignment, setting a new standard for the development and scientific evaluation of personalized LLMs.

---

References:

• "CoPA: Benchmarking Personalized Question Answering with Data-Informed Cognitive Factors" (2604.14773)