HORIZON: A Benchmark for In-the-wild User Behaviour Modeling

Abstract: User behavior in the real world is diverse, cross-domain, and spans long time horizons. Existing user modeling benchmarks however remain narrow, focusing mainly on short sessions and next-item prediction within a single domain. Such limitations hinder progress toward robust and generalizable user models. We present HORIZON, a new benchmark that reformulates user modeling along three axes i.e. dataset, task, and evaluation. Built from a large-scale, cross-domain reformulation of Amazon Reviews, HORIZON covers 54M users and 35M items, enabling both pretraining and realistic evaluation of models in heterogeneous environments. Unlike prior benchmarks, it challenges models to generalize across domains, users, and time, moving beyond standard missing-positive prediction in the same domain. We propose new tasks and evaluation setups that better reflect real-world deployment scenarios. These include temporal generalization, sequence-length variation, and modeling unseen users, with metrics designed to assess general user behavior understanding rather than isolated next-item prediction. We benchmark popular sequential recommendation architectures alongside LLM-based baselines that leverage long-term interaction histories. Our results highlight the gap between current methods and the demands of real-world user modeling, while establishing HORIZON as a foundation for research on temporally robust, cross-domain, and general-purpose user models.

• The paper presents a new benchmark that unifies 486M user-item interactions across diverse domains, addressing key limitations of prior datasets.
• The paper employs rigorous evaluation protocols, including temporal extrapolation and OOD splits, to assess model generalization.
• The paper demonstrates that both transformer-based and LLM-based approaches face challenges with long-tail, multi-domain, and temporally diverse user data.

HORIZON: A Benchmark for In-the-wild User Behaviour Modeling

Motivation and Context

HORIZON establishes a new paradigm for evaluating user modeling and sequential recommendation systems by addressing critical gaps in prior benchmarks. Traditional datasets and evaluation protocols, such as MovieLens, Amazon Reviews, and MIND, are constrained by single-domain scope, limited historical horizon, or private accessibility, and commonly focus on short-term next-item prediction under weak distribution shifts. As user behaviour on digital platforms becomes increasingly cross-domain, long-horizon, and semantically heterogeneous, benchmarks that fail to reflect these dimensions provide an incomplete assessment of model robustness and generalization. HORIZON is constructed via unified, cross-domain concatenation of Amazon Reviews 2023 [hou2024bridging], encompassing 54M users, 35M items, and nearly 486M user-item interactions. This scale not only rivals or exceeds proprietary datasets but is fully open source.

Dataset Characteristics and Analysis

HORIZON exhibits pronounced characteristics making it distinct and challenging:

1. Long-Tailed User Histories: User interaction sequence lengths follow an extremely long-tail distribution; most users have short histories, yet tens of thousands exhibit histories stretching beyond 1000 interactions, demanding architectures that model long-range dependencies and efficiently utilize memory (Figure 1).

   Figure 1: Histogram of user history lengths in HORIZON; ultra-long user histories illustrate the need for models with long-sequence capacity.

2. Temporal Diversity: The temporal distribution is balanced, with nearly equal pre- and post-2020 interactions, enabling robust temporal cut-off protocols for evaluating extrapolation and recency adaptation (Figure 2).

   Figure 2: Balanced user history temporality in HORIZON, supporting extrapolation and distribution shift analyses.

3. Item Sparsity and Extreme Long-Tail: The product frequency distribution follows a power-law with a significant fraction of items being minimally interacted with, amplifying challenges such as cold-start, novelty, and rare-item generalization for recommenders (Figure 3).

   Figure 3: Product interaction counts in HORIZON; most items have few interactions, indicating extreme item sparsity.

4. Cross-Domain Behaviour: Users on HORIZON naturally transition across product categories, with over 90% engaging multiple domains. This is in stark contrast with prior datasets, which artificially segment by category. The distribution shift between in-distribution (IND) and out-of-distribution (OOD) user cohorts reveals substantive difference in interest diversity and topical focus, as confirmed by LDA+KL analyses and t-SNE visualization (Figure 4).

   Figure 4: t-SNE plot of IND vs. OOD user topic distributions, revealing semantic and behavioral divergence.

Benchmark Task Formulations and Evaluation Protocol

The paper fundamentally rethinks evaluation, structuring tasks/methodologies around key axes of real-world generalization, rather than next-item accuracy within an artificial split.

• Task 1 — Next Item Recommendation (Explicitly Split): Rather than naïve ratio-based or leave-one-out splits, HORIZON uses a global temporal cut-off (e.g., $\tau$ = 2020) and systematically constructs 4 evaluation regimes based on user and time distribution:
  1. IND users, temporally aligned (classic leave-one-out, but with strict temporality)
  2. IND users, temporal extrapolation (all post-$\tau$ user events)
  3. OOD users, temporally aligned
  4. OOD users, extrapolation (the hardest, requiring both user and time generalization)

     Figure 5: Schematic of HORIZON’s proposed evaluation splits—disentangling user and temporal generalization axes.

• Task 2 — LLM-Based Query Reformulation: LLMs are prompted to generate sets of search queries from user histories, which are then used with ANN-retrieved item encoders (BLAIR) to assess candidate retrieval quality (Figure 6).
• Task 3 — LLM-Based Long-Horizon Modeling: LLMs are asked to autoregressively generate natural language future item descriptions; these, too, are used to retrieve candidate items, testing semantic, multi-step horizon inference.

  Figure 6: End-to-end architecture for LLM-driven user modeling: LLM prompt, ANN retrieval, and evaluation pipeline for Tasks 2 and 3.

Empirical Observations

Traditional ID-Based Sequential Models

Extensive benchmarking is provided for SOTA sequential architectures (BERT4Rec, SASRec, GRU4Rec, CORE) under HORIZON's challenging splits:

• In-distribution evaluation (Task 1a) exhibits substantially higher performance (e.g., Recall@100 near 46.6 for SASRec) compared to all OOD or temporal extrapolation splits, manifesting metrics collapse (e.g., Recall@100 ≈ 11 in worst cases for OOD/Temporal extrapolation).
• Recurrent models (GRU4Rec) that succeed in short-horizon, domain-segmented setups underperform drastically; only flexible attention-based architectures (BERT4Rec, SASRec) somewhat retain robustness, though with signficant degradation.
• Models generalize better to new users in the same period than to old users across new time periods. This highlights that temporal distributional shift is even harsher than user OOD.

LLM-Based Pipelines

• Zero-shot instruction-tuned LLMs (LLaMA-3.1-8B, Qwen3-8B, Gemma2-9B) achieve only marginal Recall/Precision (Recall@100 < 4%) for query reformulation, with Qwen3-8B best but still significantly below practical threshold even for large candidate set sizes.
• Autoregressive long-horizon description modeling slightly increases Recall@100 yet Precision does not scale, indicating retrieval of some relevant items but persistent semantic drift.
• Even with parameter-efficient and full-instruction fine-tuning, LLMs do not substantially outperform zero-shot—fine-tuning yields near-parity with vanilla prompting on next-item retrieval.
• Large reasoning models (e.g., Qwen3-235B), as per ablation, do not close the gap, and scaling model or prompt quality does not overcome the distribution gap.

Discussion and Implications

HORIZON reveals that high in-distribution metrics in conventional settings overstate both the robustness and real-world efficacy of current recommendation models. The unified, temporally rigorous, and behavioral-heterogeneity-stressing splits in HORIZON surface massive performance decays not visible in classical protocols.

Theoretical Implications

• Past approaches that decouple training/evaluation from cross-domain and temporal OOD conditions are vulnerable to “shortcut” learning: exploiting item co-occurrence or shallow session context, rather than constructing genuinely generalizable user representations.
• Transformer-based models possess some, but not sufficient, inductive bias for cross-domain temporal generalization; the absence of semantic grounding means that item or user novelty consistently erodes ranker performance.
• Prompt/sequence generation LLMs, even after extensive pretraining, cannot readily learn to retrieve or predict in extreme long-tail, unified-domain, or extrapolated settings without explicit negative sampling or alignment with item vocabularies.

Practical Implications and Future Directions

• Industry-relevant recommender evaluation requires explicitly multi-domain, temporally extrapolative, and user OOD tests such as those in HORIZON.
• Model design must incorporate enhanced semantic enrichment (e.g., textual features, cross-modal encoders), long-memory architectures, and explicit alignment mechanisms to catalog evolution.
• LLM-based approaches will require retrieval augmentation, contrastive learning, or hybrid generative-discriminative objectives to cope with item sparsity and catalog expansion. Purely generative or prompt-based approaches are insufficient in extreme long-tail recommendation.

Future advancements may include incorporating nonlinear user preference evolution models, integrating multimodal product information (text, image, audio) for cross-domain reasoning, and leveraging structured negative sampling alongside large language encoders. The extension beyond text and English-only data, as HORIZON authors note, is also a clear next step.

Conclusion

HORIZON sets a rigorous new standard for user modeling benchmark design, conclusively demonstrating that robust, real-world generalization in recommendation cannot be credibly assessed via traditional in-distribution, domain-segmented, or temporally static protocols. The empirical results expose the limitations of both classic and LLM-based recommender frameworks under realistic conditions and highlight the pressing need for new architectures and training paradigms that are fundamentally grounded in semantic, temporal, and behavioral diversity.