RecSys Challenge 2025: Universal Behavioral Modeling

• RecSys Challenge 2025 is a competition that benchmarks universal user modeling by generating task-independent embeddings from diverse user events.
• The challenge employs multi-view methods such as sequential autoencoding, collaborative filtering, and graph neural networks to capture dynamic behavioral profiles.
• It emphasizes novelty and diversity in evaluation, promoting robust performance across six downstream tasks beyond repetitive static recommendations.

Searching arXiv for papers on RecSys Challenge 2025 and closely related evaluation/task-formulation work. RecSys Challenge 2025 was a recommender-systems competition organized by Synerise and ACM RecSys that focused on universal behavioral modeling: the objective was to generate user embeddings effective across six diverse downstream tasks (Makeev et al., 9 Aug 2025). In the challenge framing described by the top-ranked solution papers, participants were asked to construct “Universal Behavioral Profiles” from logs of past user behavior, with user histories spanning product purchases, add/remove from cart events, page visits, and search queries (Klenitskiy et al., 11 Aug 2025). The challenge is notable for centering task-agnostic user representation learning rather than a single recommendation target, thereby situating itself at the intersection of industrial recommender-system practice, dynamic user modeling, and representation transfer across multiple predictive objectives (Dong et al., 2023).

1. Challenge definition and problem setting

RecSys Challenge 2025 was defined around the construction of a universal user representation that encapsulates a user’s full behavioral history and remains effective across diverse prediction tasks (Klenitskiy et al., 11 Aug 2025). The goal, as stated by the second-place solution, was to develop such Universal Behavioral Profiles from logs of past user behavior, which included various types of events such as product purchases, page views, and search queries (Klenitskiy et al., 11 Aug 2025). The fourth-place solution describes the same objective as generating user embeddings effective across six diverse downstream tasks (Makeev et al., 9 Aug 2025).

The challenge data model, as reported in the solution papers, was event-centric rather than matrix-centric. User histories were formed from heterogeneous interaction streams including cart additions, purchases, cart removals, page visits, and search queries, and these were merged chronologically for modeling (Makeev et al., 9 Aug 2025). The second-place solution gives a more explicit preprocessing view: multiple event types were merged into a single chronological sequence ordered by time; missing fields were assigned a special “missing” value; rare SKU/URL were mapped to a “rare” category; temporal information was discretized; and “start-of-sequence” and “end-of-sequence” tokens were appended for sequence boundary detection (Klenitskiy et al., 11 Aug 2025).

This formulation places the challenge closer to dynamic decision-process modeling than to static matrix completion. A closely related critique in recommender-systems research argues that recommendation is often oversimplified as filling in missing values in a static user-item interaction matrix, whereas practical systems must predict a user’s next interaction or decision within a dynamic, changing, and application-specific context (Sun, 2024). This suggests that RecSys Challenge 2025 can be read as part of a broader movement from static preference estimation toward behavior modeling over temporally ordered, context-bearing event streams.

2. Tasks, scoring, and leaderboard structure

The challenge evaluation comprised six downstream tasks. Three were visible: churn prediction, category propensity, and product propensity; three were hidden: conversion, new product propensity, and price propensity (Klenitskiy et al., 11 Aug 2025). The fourth-place solution presents the same six-task structure with slightly different naming at the revealed-task level: churn prediction, purchased category prediction, purchased item (SKU) prediction, plus three hidden tasks—conversion, price propensity, and product propensity (Makeev et al., 9 Aug 2025). Both descriptions agree that the official benchmark was multi-task and that evaluation was performed server-side on downstream tasks rather than by direct scoring of the embeddings alone (Makeev et al., 9 Aug 2025).

For category and product tasks, the second-place solution reports the scoring rule

$$\text{Score} = 0.8 \times \text{AUROC} + 0.1 \times \text{Novelty} + 0.1 \times \text{Diversity}$$

(Klenitskiy et al., 11 Aug 2025). The challenge also used a leaderboard aggregation rule based on Borda count: for $N$ teams, the $k$-th place in a task earned $N-k$ points, summed across all tasks (Klenitskiy et al., 11 Aug 2025). Performance was computed for 1M “relevant” clients, defined there as users with long histories (Klenitskiy et al., 11 Aug 2025).

The leaderboard outcomes reported in the solution papers indicate both competitiveness and sensitivity to representation design. Team ai_lab_recsys achieved second place overall and first on the academic leaderboard, with a sum of scores of 4.7504 and a Borda count of 653 (Klenitskiy et al., 11 Aug 2025). Team ambitious placed fourth overall among all teams (Makeev et al., 9 Aug 2025). These outcomes underscore that the benchmark rewarded not a single-task specialist but a representation with broad transfer across revealed and hidden tasks.

A plausible implication is that the inclusion of novelty and diversity alongside AUROC aimed to discourage purely repetitive recommendation behavior. That interpretation is consistent with prior work showing that real-world data often exhibits strong repetitive consumption patterns and that overall performance can be inflated by exploiting repeats, while novel item prediction remains much harder (Kolesnikov et al., 2021).

3. Universal Behavioral Profiles and representation-learning paradigms

The dominant methodological theme in RecSys Challenge 2025 was the construction of Universal Behavioral Profiles, or fixed-length task-independent user embeddings (Klenitskiy et al., 11 Aug 2025). The second-place solution’s core model transformed the entire user interaction history into a single chronological sequence and trained a GRU-based autoencoder to reconstruct this sequence from a fixed-size vector (Klenitskiy et al., 11 Aug 2025). In that architecture, each event was encoded as the sum of learned embeddings for categorical fields; stacked GRU layers processed the sequence; and the final hidden state of the last GRU layer became the user embedding vector (Klenitskiy et al., 11 Aug 2025). The decoder, also built from stacked GRUs, reconstructed event fields using teacher forcing, with cross-entropy losses summed across all event fields and all time steps:

$$L = \sum_{\text{users}} \sum_{t=1}^T \sum_{f=1}^F \text{CE}(y_{t,f}, \hat{y}_{t,f})$$

(Klenitskiy et al., 11 Aug 2025).

This sequence-autoencoding approach was supplemented by several complementary embedding methods. The same solution incorporated collaborative filtering with iALS and LightFM, transformer-based next-event prediction, language-model-based embeddings using SmolLM2-135M, and handcrafted statistical features (Klenitskiy et al., 11 Aug 2025). The final system applied PCA to high-dimensional embeddings, normalized each embedding to unit length, applied mean imputation for missing values, and concatenated all components into a 1066-dimensional universal user vector (Klenitskiy et al., 11 Aug 2025). The paper reports that the final ensemble outperformed any single embedding approach, demonstrating strong complementarity between sequential autoencoding, collaborative filtering, transformer models, and LLM-derived text embeddings (Klenitskiy et al., 11 Aug 2025).

The fourth-place solution pursued a different but similarly hybrid representation strategy. Its Universal Behavioral Profile combined four components: a sequential encoder, a graph neural network based on TwHIN, a deep cross network based on DCN-v2, and handcrafted engineered features (Makeev et al., 9 Aug 2025). The sequential encoder was a single transformer with causal attention over a heterogeneous event sequence, using content and context embeddings; the output user embedding was the hidden state of the final token (Makeev et al., 9 Aug 2025). The model was trained with multi-task token-level losses for next event type prediction, time delta prediction, next URL prediction, and next item prediction (Makeev et al., 9 Aug 2025). The final embedding concatenated the sequential encoder output, TwHIN GNN embedding, deep cross network embedding, and standardized numeric feature vector (Makeev et al., 9 Aug 2025).

Across these systems, a common design principle was heterogeneity by construction. Sequence reconstruction, next-event forecasting, collaborative structure, graph-based link prediction, feature crosses, and engineered summary statistics were treated as complementary views of behavior rather than mutually exclusive alternatives (Makeev et al., 9 Aug 2025). This suggests that RecSys Challenge 2025 functioned not merely as a modeling competition but as a benchmark for compositional user representation learning.

4. Relation to industrial recommender-system challenges

The challenge’s emphasis on universal user modeling aligns with several longstanding problems in industrial recommender systems. A 2023 synthesis from Huawei Noah’s Ark Lab identifies ten critical challenges in large-scale industrial recommender systems: missing information, individual treatment effect, biases, model reuse, LLM enhanced recommendation, multiple modalities, simulation, lifetime value modeling, trustworthy recommender systems, and a win-win ecosystem for multi-stakeholders (Dong et al., 2023).

Several of these are directly implicated by the RecSys Challenge 2025 design. The use of heterogeneous logs and sequence encoders addresses, at least partially, the problem of missing information by attempting to extract robust signals from incomplete and varied event data (Dong et al., 2023). The inclusion of search queries, page visits, product events, and temporal indices points toward multi-modality in a broad behavioral sense, even though the representations described in the provided solution papers remain fundamentally event-sequence-based (Klenitskiy et al., 11 Aug 2025). The use of novelty and diversity in scoring is closely related to the broader problem of biases and operational feedback loops in recommender systems (Dong et al., 2023).

The challenge is also relevant to lifetime value modeling. Huawei’s synthesis notes that most systems optimize short-term metrics such as clicks and dwell time, while neglecting long-term satisfaction and engagement (Dong et al., 2023). RecSys Challenge 2025 did not directly optimize lifetime value, but the task set included churn prediction and conversion, and the overall framing of task-independent profiles spanning multiple downstream objectives moves beyond single short-term engagement targets (Klenitskiy et al., 11 Aug 2025). A plausible implication is that universal behavioral representations may support downstream long-horizon objectives more naturally than narrowly optimized task-specific embeddings.

The challenge further intersects with the industrial interest in LLM-enhanced recommendation and simulation. One top solution used SmolLM2-135M to convert entire event histories, expressed as text, into embeddings (Klenitskiy et al., 11 Aug 2025). At the same time, the broader industrial challenge list identifies efficient integration, bias handling, and system scaling as open fronts for LLM-enhanced recommendation (Dong et al., 2023). Thus, the challenge can be seen as an empirical testbed for whether LLM-derived features can contribute useful complementary signal in production-style recommendation settings.

5. Evaluation realism, temporal structure, and novelty

RecSys Challenge 2025 also sits within a methodological debate about how recommender systems should be formulated and evaluated. A recent critique argues that academic recommender-systems research often conceptualizes recommendation as prediction of missing values in a static interaction matrix, neglecting the dynamic context in which users make decisions (Sun, 2024). It proposes reframing tasks as prediction of a user’s next interaction or decision given contextual features $X$, user history $I_u$, current session interactions $I_c$, the current candidate item pool $I$, and historical interactions $U \times I$ (Sun, 2024). It also argues that offline evaluation should respect global temporal order and simulate online deployment rather than relying on random splits that leak future information (Sun, 2024).

The challenge’s event-sequence formulation is consistent with that direction. Both top solutions model merged chronological histories and treat user behavior as temporally ordered heterogeneous sequences rather than as entries in a static matrix (Klenitskiy et al., 11 Aug 2025). The fourth-place solution explicitly used causal attention and selected the latest $N$0 events for inference, reinforcing a forward-only temporal interpretation (Makeev et al., 9 Aug 2025). Although the provided summaries do not explicitly state the challenge split protocol, the architectural choices and task definition are closer to dynamic behavior modeling than to traditional static collaborative filtering.

Novelty is a particularly important lens for interpreting the challenge. The official score for category and product tasks assigned nonzero weight to Novelty and Diversity in addition to AUROC (Klenitskiy et al., 11 Aug 2025). Prior benchmarking work on next-period recommendation showed that real-world datasets frequently exhibit strong repetitive consumption, that simple personalized frequency-based methods remain highly competitive, and that performance on novel-only items drops drastically for all methods (Kolesnikov et al., 2021). That work recommends evaluating both repeat and novel recommendation performance separately and using time-aware global splits to avoid data leakage (Kolesnikov et al., 2021).

Within that context, the challenge’s multi-objective scoring can be interpreted as a partial response to the “repeat-dominance” problem. This suggests that RecSys Challenge 2025 was designed not only to identify embeddings with high discriminative accuracy, but also to reward representations that support broader behavioral coverage beyond repeated-item memorization.

6. Reported results, ablations, and technical lessons

The second-place solution reports that GRU-AE models were consistently strong across all tasks and delivered the best single-model performance (Klenitskiy et al., 11 Aug 2025). An autoencoder ensemble in which a simple version using only day and event type replaced another component improved new product prediction on one hidden task (Klenitskiy et al., 11 Aug 2025). The paper’s main technical conclusion is that forcing reconstruction of the complete user event sequence creates embeddings that robustly generalize across diverse downstream tasks (Klenitskiy et al., 11 Aug 2025).

The same paper attributes much of its final performance to diversity via ensembling. Heterogeneous representations—reconstruction-based, next-item, collaborative filtering, LLM embeddings, and handcrafted statistics—captured complementary facets of user behavior (Klenitskiy et al., 11 Aug 2025). It also emphasizes practical engineering choices such as temporal bucketing, handling of high-cardinality fields, normalization, and dimensionality reduction as crucial (Klenitskiy et al., 11 Aug 2025). These observations indicate that the challenge rewarded not only architectural novelty but also feature-system design and representation calibration.

The fourth-place solution provides a detailed ablation perspective. It reports that all sequential encoder losses were critical, that removal of next-item prediction caused the largest drop, that larger transformers consistently improved downstream tasks, that inclusion of both purchases and cart-additions in TwHIN improved downstream item/category prediction, and that contrastive loss improved generalization for the DCN-based embedding (Makeev et al., 9 Aug 2025). Component-level ablations showed that removing the sequential or DCN embedding led to metric drops, while TwHIN made a marginal but positive contribution and numeric features had minor impact, likely due to redundancy with DCN (Makeev et al., 9 Aug 2025).

These findings point to a general technical lesson: universal behavioral profiling benefits from multi-view supervision and modular composition. Sequence models capture temporal evolution, graph methods inject collaborative structure, feature crosses model high-order interactions, and engineered statistics summarize stable aggregates (Makeev et al., 9 Aug 2025). Rather than displacing one another, these components appear to interact as partially overlapping sources of signal.

7. Significance and research directions

RecSys Challenge 2025 is significant because it operationalized a task-agnostic representation-learning objective in a recommender-systems benchmark. Instead of asking for optimization of a single ranking task, it evaluated whether a single user embedding could transfer across churn prediction, propensity estimation, conversion, price propensity, and related objectives (Klenitskiy et al., 11 Aug 2025). In that respect, it reoriented challenge design toward universal user modeling and away from narrowly scoped target prediction.

This orientation resonates with the broader claim that recommender-systems research should be more closely aligned with real-world application scenarios, richer input features, temporal dynamics, and scenario-specific evaluation (Sun, 2024). It also parallels industrial concerns that future recommendation systems will require holistic invention at the intersection of technology, business, and society, rather than better algorithms in isolation (Dong et al., 2023).

At the same time, the challenge does not resolve the core open problems identified in the surrounding literature. Missing data, exposure bias, individual treatment heterogeneity, efficient model reuse, robust LLM integration, simulation fidelity, long-term value optimization, trustworthiness, and multi-stakeholder balancing remain open research fronts (Dong et al., 2023). Novel item prediction likewise remains difficult under realistic temporal evaluation, especially in domains with strong repetitive behavior (Kolesnikov et al., 2021).

A plausible implication is that RecSys Challenge 2025 marks a shift in benchmark design rather than an endpoint in methodology. Its central contribution lies in making universal behavioral modeling a concrete competitive target, thereby encouraging architectures and evaluation strategies that better reflect the temporal, heterogeneous, and multi-objective character of modern recommender systems (Makeev et al., 9 Aug 2025).