GISTBench: Evaluating LLM User Understanding via Evidence-Based Interest Verification

Abstract: We introduce GISTBench, a benchmark for evaluating LLMs' (LLMs) ability to understand users from their interaction histories in recommendation systems. Unlike traditional RecSys benchmarks that focus on item prediction accuracy, our benchmark evaluates how well LLMs can extract and verify user interests from engagement data. We propose two novel metric families: Interest Groundedness (IG), decomposed into precision and recall components to separately penalize hallucinated interest categories and reward coverage, and Interest Specificity (IS), which assesses the distinctiveness of verified LLM-predicted user profiles. We release a synthetic dataset constructed on real user interactions on a global short-form video platform. Our dataset contains both implicit and explicit engagement signals and rich textual descriptions. We validate our dataset fidelity against user surveys, and evaluate eight open-weight LLMs spanning 7B to 120B parameters. Our findings reveal performance bottlenecks in current LLMs, particularly their limited ability to accurately count and attribute engagement signals across heterogeneous interaction types.

• The paper introduces GISTBench, a novel benchmark evaluating LLM-based user understanding by verifying interests against behavioral evidence.
• It measures Interest Groundedness and Specificity, decoupling user modeling from recommendation accuracy with robust evidence attribution and taxonomy normalization.
• Experimental results reveal coverage bottlenecks in LLMs, emphasizing the need for enhanced multi-hop reasoning and long-context evidence aggregation.

GISTBench: A Benchmark for Evidence-Based User Understanding in LLMs

Problem Definition and Motivation

GISTBench introduces an explicit evaluation paradigm for User Understanding in RecSys, operationalized for LLMs through evidence-based interest verification. Unlike traditional RecSys tasks that focus on next-item or click prediction, GISTBench targets the model’s ability to produce verifiable, natural-language user interest profiles derived exclusively from interaction history, decoupling user modeling from recommendation accuracy. This reframing is essential for generative pipeline systems, personalized conversational agents, and cross-modal profile portability, which all require interpretable, auditable user models.

Current benchmarks inadequately measure groundedness: item prediction confounds user interests with positional/temporal/contextual biases, and profile plausibility is orthogonal to factual grounding. GISTBench corrects these deficiencies by directly measuring groundedness (verifiable from behavioral logs) and specificity (distinctiveness at the category level), establishing new evaluation primitives tailored to the LLM context.

Figure 1: The GISTBench pipeline processes user histories, extracts predicted interests via LLMs, and computes Interest Groundedness/Specificity, normalizing to a standardized taxonomy.

Dataset Construction and Taxonomy Normalization

GISTBench comprises a synthetic, privacy-preserving dataset sourced from anonymized and aggregated real-world short-form video platform interaction logs, spanning implicit/explicit positive and implicit negative engagement signals. Synthetic user histories are generated through four steps: cohort formation by interest embeddings, aggregated/shuffled UIH pooling, random engagement samplings, and content description generation via a proprietary VLM. This methodology provides dense, diverse, and privacy-compliant samples without direct user linking.

Complementary evaluation is performed on KuaiRec, MIND, Amazon Music, and Goodreads (Table~\ref{tab:dataset_comparison}). Each dataset poses unique challenges in terms of interaction signal variety, textual richness, and sparsity.

All interests are normalized to a hierarchical taxonomy (325 categories) pre-defined for video, supporting large-scale inter-model comparison, avoiding depth-breadth conflation, and ensuring semantic aggregation. Taxonomy mapping employs batch LLM classification with manual audit.

Metric Families: Interest Groundedness and Specificity

Interest Groundedness (IG)

IG is formulated as a compositional metric family, explicitly decomposing precision (IG$_P$) from recall (IG$_R$), with scores aggregated at the category level after LLM judge-based evidence filtering. Interests must be supported by multi-instance, semantically relevant engagement signals (with per-signal-configurable thresholds) and not overruled by excessive contradictory evidence.

• Precision (IG$_P$): Fraction of predicted categories with at least one verified interest (no hallucination).
• Recall (IG$_R$): Fraction of "oracle" discoverable categories present (coverage).
• F1 (IG$_{F1}$): Harmonic mean, capturing joint performance.

The verification pipeline (closed-loop via instruction and judge prompt) enforces hard evidence thresholds, rendering claims empirically auditable and preventing format hackable hallucinations.

Interest Specificity (IS)

IS quantifies the discriminability of verified interests by measuring, through an LLM judge, whether a predicted interest allows retrieval of its supporting interaction objects from a large distractor pool. IS is only computed on verified categories—unverified (ungrounded/hallucinated) interests are not rewarded—bounding specificity evaluation to true inferences.

IS is robust to interest granularity, penalizing overly generic predictions incapable of accurate retrieval and rewarding correct fine-grained attributions.

Figure 3: IG Precision vs.\ IG Recall per model on the survey dataset, with bubble size encoding average category count and color encoding IG$_{F1}$.

Experimental Results and Analysis

Eight open-weight LLMs (7B–235B, spanning Llama, Qwen, DeepSeek, GPT-OSS families) are benchmarked across five datasets. All inference is at temperature 0 for strict reproducibility, with structured output parsing, and robust evidence attribution.

Figure 2: Per-user IG Precision vs.\ IG Recall across datasets, highlighting model and user-level performance variance. The $y=x$ diagonal marks the typical IG$_P >$ IG$_R$ asymmetry.

Key Numerical Findings

• Best IG$_{F1}$: GPT-OSS-120B achieves top IG$_R$0 (up to 67.8% synthetic, 66.7% MIND, 66.7% Amazon Music).
• Best IS: DeepSeek-R1 leads on specificity (up to 79.0% synthetic).
• Performance Bottleneck: All capable models consistently exhibit substantially higher IG$_R$1 than IG$_R$2; hallucination is rare but coverage is limited.
• Model Scale Dependence: Smaller models (Llama-3.1-8B, Qwen2-7B) are format-incompetent; precision/recall collapse to near zero.
• Instruction Following Ceiling: Sufficient instruction-following, as measured by Arena IF Elo, is a necessary but insufficient condition for user understanding—the task requires domain/judge reasoning not captured by standard chatbot alignment.
• Evidence Counting: Insufficient positive signal attribution is the dominant failure mode across all unverified interests; excessive negative contamination is less prevalent and model-class specific.

Figure 4: Forest plot: 95% confidence intervals for mean IG$_R$3 by model, illustrating statistically robust stratification between capable and weak models.

Figure 5: Per-model taxonomy coverage. All but Llama-3.1-8B populate $_R$497.5\% of the 325 standardized categories; Llama-3.1-8B covers under half.

Figure 6: Cumulative distribution of predicted interests across taxonomy categories, evidencing a heavy tail: 10\% of categories comprise 50\% of mappings (both dataset and model driven).

Implications and Theoretical Considerations

GISTBench's decoupled metric families demonstrate that evidence-based interest verification poses unique reasoning challenges for LLMs, not reducible to instruction following or plausibility. The precision-recall decomposition reveals that interest profile accuracy for downstream generative user modeling is bottlenecked almost entirely by correct evidence attribution and aggregation, not hallucination suppression.

The long-tail distribution of interests, coupled with the necessity of multi-modal aggregation (day-long video interactions, behavioral only), means LLMs require both adequately long context windows and algorithmically-targeted attention/aggregation strategies. This result calls into question the sufficiency of standard instruction tuning and format alignment for recsys-style grounded user understanding.

From a theoretical standpoint, the GISTBench framework affirms that grounding metrics capturing both faithfulness (from evidence) and plausibility (from distinguishability) are essential for generative recommender compliance—a necessary precondition before LLM profiles can be trusted for on-device explanation, cross-platform transfer, or policy-sensitive personalization.

Practical Directions and Future Research

• Modeling: Demonstrably, performance is limited by both reasoning scale and signal attribution. Multi-hop attention over heterogeneous signals, and possibly novel memory-augmented architectures, are required to move recall towards human fidelity.
• Reward Shaping: GISTBench's IG and IS can be integrated as targeted RLHF objectives for fine-tuning, penalizing coverage failure, and ungrounded generality beyond instruction-compliance.
• Multimodality: Extension to multimodal UIHs—including visual and audio cues—could directly challenge current LLMs' inherent modality bottlenecks.
• Synthetic Persona Alignment: Incorporating persona-grounded construction (cf. "Scaling synthetic data creation with 1,000,000,000 personas" (Ge et al., 2024)) enables evaluation of alignment between explicit user identities and behavioral-based profiles.
• Oracle Robustness: The union-based oracle for IG$_R$5 is inherently model-dependent, calling for standardization around ensemble or human-aggregated upper bounds in longitudinal benchmarks.

Limitations

GISTBench's current instantiation is domain- and taxonomy-dependent. Verification thresholds, while correlated with ground-truth survey data ($_R$6), may require recalibration across other content modalities, cultures, or recsys application classes. Model comparability for IG$_R$7 is valid only within fixed ensemble settings. All judge filtering and retrieval is currently LLM-based (Llama-3.3-70B-Instruct); cross-judge generalizability remains an open ablation.

Conclusion

GISTBench establishes a new paradigm for LLM-based evaluation of user understanding, prioritizing evidence-based verification and discriminative specificity. The framework exposes coverage—not hallucination—as the persistent failure mode for current open-weight models. Progress towards reliable, explainable, and user-aligned recommendation pipelines will require advancements in long-context evidence aggregation, reasoning specialization, and evaluation protocols grounded in empirical profile verifiability.